Hyperspectral Image Classification—Traditional
to Deep Models: A Survey for Future Prospects

Muhammad Ahmad,
Sidrah Shabbir,
Swalpa Kumar Roy, Student Member, IEEE,
Danfeng Hong, Senior Member, IEEE,
Xin Wu, Member, IEEE,
Jing Yao,
Adil Mehmood Khan,
Manuel Mazzara,
Salvatore Distefano,
and Jocelyn Chanussot Fellow, IEEE

Manuscript received October 24, 2021; revised November 19, 2021; accepted
November 30, 2021. Date of publication December 9, 2021; date of current
version January 20, 2022. This work was supported in part by the National Natural Science Foundation of China under Grant 42030111 and Grant 41722108. This work was supported by the National Natural Science Foundation of China under Grant 62101045 and the China Postdoctoral Science Foundation Funded Project No. 2021M690385. This work was supported by MIAI@Grenoble Alpes (ANR-19-P3IA-0003) and the AXA Research Fund. This research was also financially supported by The Analytical Center for the Government of the Russian Federation (Agreement No. 70-2021-00143 dd. 01.11.2021, IGK 000000D730321P5Q0002) Corresponding author: Danfeng Hong.
M. Ahmad is with Department of Computer Science, National University of Computer and Emerging Sciences, Islamabad, Chiniot-Faisalabad Campus, Chiniot 35400, Pakistan, and Dipartimento di Matematica e Informatica—MIFT, University of Messina, Messina 98121, Italy; (e-mail<EMAIL_ADDRESS>S. Shabbir is with the Department of Computer Engineering, Khwaja Fareed University of Engineering and Information Technology (KFUEIT), Pakistan. (e-mail<EMAIL_ADDRESS>S. K. Roy is with the Department of Computer Science and Engineering, Jalpaiguri Government Engineering College, West Bengal 735102, India (e-mail: swalpa@cse.jgec.ac.in).
D. Hong and J. Yao are with the Key Laboratory of Digital Earth Science, Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China (e-mail<EMAIL_ADDRESS>yaojing@aircas.ac.cn).
X. Wu is with the School of Information and Electronics, Beijing Institute of Technology, 100081 Beijing, China, and Beijing Key Laboratory of Fractional Signals and Systems, 100081 Beijing, China. (e-mail<EMAIL_ADDRESS>A. M. Khan is with the Institute of Data Science and Artificial Intelligence, Innopolis University, Innopolis, 420500, Russia. (e-mail<EMAIL_ADDRESS>M. Mazzara is with Institute of Software Development and Engineering, Innopolis University, Innopolis, 420500, Russia. (e-mail<EMAIL_ADDRESS>S. Distefano is with  Dipartimento di Matematica e Informatica—MIFT, University of Messina, Messina 98121, Italy. (e-mail<EMAIL_ADDRESS>J. Chanussot is with the Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France. (e-mail<EMAIL_ADDRESS>Digital Object Identifier 10.1109/JSTARS.2021.3133021

IEEE JOURNAL OF SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, VOL. 15, 2022
M.Ahmad et al.:

Hyperspectral Imaging (HSI) has been extensively utilized in many real-life applications because it benefits from the detailed spectral information contained in each pixel. Notably, the complex characteristics i.e., the nonlinear relation among the captured spectral information and the corresponding object of HSI data make accurate classification challenging for traditional methods. In the last few years, Deep Learning (DL) has been substantiated as a powerful feature extractor that effectively addresses the nonlinear problems that appeared in a number of computer vision tasks. This prompts the deployment of DL for HSI classification (HSIC) which revealed good performance. This survey enlists a systematic overview of DL for HSIC and compared state-of-the-art strategies of the said topic. Primarily, we will encapsulate the main challenges of traditional machine learning for HSIC and then we will acquaint the superiority of DL to address these problems. This survey breakdown the state-of-the-art DL frameworks into spectral-features, spatial-features, and together spatial-spectral features to systematically analyze the achievements (future research directions as well) of these frameworks for HSIC. Moreover, we will consider the fact that DL requires a large number of labeled training examples whereas acquiring such a number for HSIC is challenging in terms of time and cost. Therefore, this survey discusses some strategies to improve the generalization performance of DL strategies which can provide some future guidelines.

Hyperspectral Imaging (HSI), Hyperspectral Image Classification (HSIC), Deep Learning (DL), Feature Learning, Spectral-Spatial Information.

§ INTRODUCTION

Hperspectral Imaging (HSI) is concerned with the extraction of meaningful information based on the radiance acquired by the sensor at short or long distances without substantial contact with the object of interest [1]. HSI provides detailed spectral information by sampling the reflective portion of the electromagnetic spectrum covering a wide range of $0.4-2.4~m$ (i.e. visible $0.4-0.7~m$ to short wave infrared $0.7-2.4~m$) region in hundreds of narrow and contiguous spectral bands. HSI can also explore the (light) emission properties of objects in the range of mid to long infrared regions [2].

Various real-world applications of HSI.

Despite the detailed information, it brings several challenges since traditional analysis techniques for monochromatic, RGB, and multispectral images cannot be directly exploited to extract meaningful information from Hyperspectral ones due to several reasons, e.g. HSI exhibits the unique statistical and geometrical properties of high dimensional spectral/spatial data, i.e. the volume of a hypercube and hypersphere concentrates on corners and outside shells respectively.

HSI has been adopted in several real-world applications including but not limited to the atmosphere, environmental, urban, agriculture, geological and mineral exploration, coastal zone, marine, forestry (i.e. track forest health), water quality and surface contamination, inland waters, and wetlands, snow and ice, biological, medical contexts, and food processing [3, 4, 5, 6, 7, 8]. There are also several military applications in camouflage, landmine detection, and littoral zone mapping. Furthermore, HSI has been used in space, air, and underwater vehicles to acquire detailed spectral information for a wide range of uses [9, 10, 11, 12].

Infield collection and spectral library indexing of ground truth signatures for any of the said applications are critical for many reasons. For instance, the spectral information of vegetation is prejudiced by a wide range of environmental situations that make it challenging to satisfactorily represent variability without the collection of site-specific field spectra. But the real potential of HSI is mostly untapped since it allows it to go deeper than surface features considering that usually, each feature has a different spectrum band. HSI, indeed, can capture more than 200 spectral bands which help practitioners to discriminate objects that were not possible before. A few HSI application examples are shown in Fig. <ref>, but several other domains (e.g. smart city, Industry 4.0, Intelligent Transportation Systems) can greatly benefit from such an approach.

Considering the aforementioned limitations, HSI analysis is categorized into the following main streams: dimensionality reduction [13, 14, 15, 16, 17], spectral unmixing [18, 19, 20, 21, 22, 23, 24, 25, 26], object/change detection [27, 28, 29, 30, 31, 32, 33] classification [34, 35, 36], feature learning for classification [37, 38, 39, 40, 41], restoration and denoising [42, 43], resolution enhancement [44, 45]. Figure <ref> shows an exponentially growing trend in literature published per year for HSI analysis-related tasks and applications.

Various HSI related articles published per year till September 25, 2021, [Source: Google Scholar accessed on September 25, 2021 and the results (including patents and citations) were sorted by relevance].

In this survey, we specifically focus on HSI data classification (HSIC), which has achieved a phenomenal interest of the research community due to its broad applications in the areas of land use and land cover [46, 47, 48, 49, 50], environment monitoring and natural hazards detection [51, 52], vegetation mapping [53, 54] and urban planning. HSIC methodologies exploit machine learning algorithms to perform the classification task [55, 56]. These methods are outlined in various comprehensive reviews published during/in the last decade [57, 58, 59, 34, 60, 61, 62, 63, 64, 65]. Nevertheless, continuous advancements in the field of Machine Learning provide improved methods from time to time. Deep learning (DL) models is one of such revolutionary advancements in machine learning that improved HSIC accuracy [66, 67, 68].

This survey aims to give an overview of the widely used DL-based techniques to perform HSIC. Specifically, we will first summarize the main challenges of HSIC which cannot be effectively overcome by traditional machine learning (TML), and later we will enlist the advantages of DL to handle the aforementioned issues. At a later stage, we will provide a framework to categorize the corresponding works among:

* Spectral and spatial feature learning, individually, and

* Spectral-spatial feature learning to systematically review the achievements in DL-based HSIC.

* Future research stems to improve the generalization performance and robustness of DL models while considering the limited availability of reliable training samples.

The remainder of this paper is structured as follows.
Section <ref> introduces the task of HSI Classification (HSIC) and briefly discusses the HSIC paradigm shift from Traditional (Conventional) Machine Learning to Deep Learning (DL) models, describing HSI data characteristics along with the advantages and limitations of DL that are faced while working with HSI. In section <ref> and <ref>, we give an overview of different forms of HSI representations and basic machine learning strategies, respectively. Section <ref> describes a few commonly used types of layers and reviews recent developments (specifically from 2017 onward) of some intensively utilized DL frameworks for HSIC. Sections <ref>, <ref>, <ref>, and <ref> presents the state-of-the-art developments of Convolutional Neural Networks (CNN), Graph CNN (GCNN), Autoencoders (AEs), Deep Belief Networks (DBNs), Recurrent Neural networks (RNNs), respectively. In section <ref>, we briefly discussed various strategies to overcome the low generalization performance of HSIC due to the limited availability of training data. Section <ref> presents the experimental results and discussion on results obtained using different deep learning strategies. Section <ref> concludes the paper with a few future research directions related to joint exploitation of spectral-spatial features of HSI, limited training data, and computational complexity.

§ HYPERSPECTRAL IMAGE CLASSIFICATION (BACKGROUND AND CHALLENGES)

§.§ Traditional to DL Models

The main task of HSIC is to assign a unique label to each pixel vector of HSI cube based on its spectral or spectral-spatial properties. Mathematically, an HSI cube can be represented as \(\textbf{X} = [x_1, x_2, x_3, \dots, x_B]^T \in \mathcal{R}^{B \times (N \times M)}\), where \(B\) represent total number of spectral bands consisting of \((N \times M)\) samples per band belonging to \(\textbf{Y}\) classes where \(x_i = [x_{1,i},~x_{2,i},~x_{3,i}, \dots,x_{B,i}]^T\) is the \(i^{th}\) sample in the HSI cube with class label \(y_i \in \mathcal{R}^Y \). The classification problem can be considered as an optimization one, in which a mapping function \(f_c(.)\) takes the input data \(\textbf{X}\) and after applying some transformations over it, obtains the corresponding label \(\textbf{Y}\), to reduce the gap between obtained output and the actual one [69].

\begin{equation}
Y = f_c(X,\theta)
\end{equation}

where $\theta$ is a certain adjustable parameter that may be required to apply transformations on input data $\textbf{X}$ such that $f_c: X \to Y$.

In literature, substantial work has been done on HSIC and there is a growing trend in the development of such techniques as shown in Figure <ref>. Most HSIC frameworks seemed to be influenced by the methodologies used in the computer vision domain [70]. Traditional machine learning-based HSIC approaches use hand-crafted features to train the classifier. These methods generally rely on utilizing engineering skills and domain expertise to design several human-engineered features, for instance, shape, texture, color, shape, spectral and spatial details. All these features are basic characteristics of an image and carry effective information for image classification. Commonly used hand-crafted feature extraction and classification methods include: texture descriptors such as Local Binary Patterns (LBPs) [71], Histogram of Oriented Gradients (HOG) [72], Global Image Scale-invariant Transform / Global Invariant Scalable Transform (GIST) [73], Pyramid Histogram of Oriented Gradients (PHOG), Scale-invariant Feature Transform (SIFT) [74], Random Forests [75], kernel-based Support Vector Machine (SVM) [76], K-nearest Neighbours (KNN), and Extreme Learning Machine (ELM).

Remote sensing/Hyperspectral Image Classification related articles published per year till September 25, 2021, [Source: Google Scholar accessed on September 25, 2021 and the results (including patents and citations) were sorted by relevance].

Color histograms are simple and effective handcrafted features used for an image classification task. They are easy to compute and invariant to small changes in images i.e. translation and rotation. The major drawback of a color histogram is that it does not provide spatial contextual information, hence it becomes difficult to distinguish between objects of the same color but different distribution. Moreover, color histograms are sensitive to variance in illumination. HOG features represent the histogram of edge orientations of spatial sub-regions. It can effectively extract the edge and local shape details and has been utilized in various remote sensing related works [77, 78, 79, 46].

Scale-invariant Feature Transform (SIFT) is a broadly used robust feature descriptor applied to image classification tasks [80, 81, 82, 83]. The advantage of the SIFT descriptor is that it is invariant to the changes in image scale, rotation, illumination, and noise. SIFT is used to extract local features that describe a specific point in the image. The disadvantage of SIFT is that it is mathematically complex which increases its computational cost. GIST represents the global description of important aspects of an image that is the scales and orientations (gradient information) of various subregions of an image. GIST builds a spatial envelope in terms of different statistical properties like roughness, openness, and ruggedness, etc [84]. Texture descriptors such as local binary patterns (LBPs) are used for remote sensing image analysis [71, 85]. LBPs are used to describe the texture around each pixel by choosing pixels from the square neighborhood and gray level values of all neighborhood pixels are thresholded with respect to the central pixel.

The color histograms, GIST, and texture descriptors are global features that represent certain statistical characteristics of an image like color, texture [86, 87], and spatial structure [73]. While HOG and SIFT are local features that describe geometrical information. Usually they are used to construct bag-of-visual-words (BoVW) models [52, 88, 48, 89, 90, 91, 92, 83, 93] and HOG feature-based models [46, 94]. Some popular feature encoding or pooling strategies to enhance the performance of BoVW are Fisher vector coding [95, 96, 71], Spatial Pyramid Matching (SPM) [97], and Probabilistic Topic Model (PTM) [98, 99, 100, 93]. A single feature is insufficient to represent the whole image information, hence a combination of these features is used for image classification [101, 47, 88, 98, 100, 102, 103, 104, 105, 106, 107].

Hand-crafted features can effectively represent the various attributes of an image, hence working well with the data being analyzed. However, these features may be insubstantial in the case of real data, therefore it is difficult to fine-tune between robustness and discriminability as the set of optimal features considerably vary between different data. Furthermore, human involvement in designing the features considerably affects the classification process, as it requires a high level of domain expertise to design hand-crafted features.

To mitigate the limitations of hand-crafted feature designing, a deep feature learning strategy was proposed by Hinton and Salakhutdinov in $2006$ [108]. Deep learning (DL) based methods can automatically learn the features from data in a hierarchical manner, to construct a model with growing semantic layers until a suitable representation is achieved. Such models have shown great potential for feature representation in remote sensing image classification [109, 110].

DL architectures can learn the behavior of any data without any prior knowledge regarding the statistical distribution of the input data [111] and can extract both linear and non-linear features of input data without any pre-specified information. Such systems are capable of handling HSI data in both spectral and spatial domains individually, and also in a coupled fashion. DL systems possess a flexible architecture in terms of types of layers and their depth and are adaptive to various machine learning strategies like supervised, semi-supervised, and unsupervised techniques.

§.§ Hyperspectral Data Characteristics and DL Challenges

Despite the above-discussed DL potentials, there are still some challenges that need to be considered while applying DL to HSI data. Most of these challenges are related to the characteristics of HSI data i.e. hundreds of contiguous and narrow spectral channels with very high spectral resolution and low spatial resolution throughout the electromagnetic spectrum coupled with limited availability of training data. Although the pixels with rich spectral information are useful for classification purposes, however, the computation of such data takes a lot of time and resources.

Furthermore, processing such high-dimensional data is a somewhat complex task due to an increased number of parameters. This is known as the curse of dimensionality which considerably influences the classification performance especially in the case of supervised learning [112]. Since the size of training data is not adequate/insufficient and/or not reliable (i.e. the training samples may not provide any new information to the model or may have similar patterns/structures) to properly train the classifier which may lead the model to overfit. This is known as the Hughes phenomena [113] which occurs when labeled training data is significantly smaller than the number of spectral bands present in the data. Lack of labeled HSI data is a major issue in HSIC as labeling of HSI is a time-consuming and expensive task because it usually requires human experts or investigation of real-time scenarios.

In addition to high dimensionality, HSIC suffers from various other artifacts like high intra-class variability due to unconfined variations in reflectance values caused by several environmental interferers and degradation of data caused by instrumental noise while capturing the data [114]. Furthermore, the addition of redundant bands due to HSI instruments affects the computational complexity of the model. Spectral mixing is another challenge related to the spatial resolution of HSI. HSI pixels with low to average spatial resolution cover vast spatial regions on the surface of earth leading to mixed spectral signatures which result in high inter-class similarity in border regions. As a result, it becomes difficult to identify the materials based on their spectral reflectance values [115]. Following are some main challenges that come across when DL is applied to HSIC:

* Complex Training Process: Training of Deep Neural Network (DNN) and optimization by tuning parameters is an NP-complete problem where the convergence of the optimization process is not guaranteed [116, 117, 118]. Therefore, it is assumed that training of DNN is very difficult [111] especially in the case of HSI when a large number of parameters need to be adjusted/tuned. However, the convergence task becomes somehow easier due to the advancement of various optimization techniques for deep CNNs. Among stochastic gradient descent (SGD) [119] and its momentum version (SGDM) [120], RMSProp [121], Adam [122], AdamW [123], diffGrad [124], RAdam [125], gradient centralization (GC) [126], AngularGrad [127], respectively are the successful CNN optimization techniques and widely used in any classification problems.

* Limited Availability of Training Data: As discussed above, supervised DNN requires a considerably large amount of training data otherwise their tendency to overfit increases significantly [128] leads to the Hughes phenomena. The high dimensional characteristic of HSI coupled with a small amount of labeled training data makes the DNNs ineffective for HSIC as it demands a lot of adjustments during the training phase [69].

* Model's Interpretability: The training procedure of DNNs is difficult to interpret and understand. The black box kind of nature is considered as a potential weakness of DNNs and may affect the design decisions of the optimization process. Although, a lot of work has been done to interpret the model's internal dynamics.

* High Computational Burden: One of the main challenges of DNN is dealing with a big amount of data that involves increased memory bandwidth, high computational cost, and storage consumption [129]. However, advanced processing techniques like parallel and distributed architectures [130, 131] and high-performance computing (HPC) [115] make it possible for DNNs to process large amounts of data.

* Training Accuracy Degradation: It is assumed that deeper networks extract more rich features from data [132], however, this is not true for all systems to achieve higher accuracy by simply adding more layers. Because by increasing the network’s depth, the problem of exploding or vanishing gradient becomes more prominent [133] and affects the convergence of the model [132].

§ HSI REPRESENTATION

Hyperspectral data is represented in the form of a $3D$ hypercube, $X \in \mathcal{R}^{B \times (N \times M)}$,  which contains $1D$ spectral and $2D$ spatial details of a sample where $B$ represents the total number of spectral bands and $N$ and $M$ are spatial components i.e., width and height, respectively. The HSI cube is shown in Figure <ref>.

Hyperspectral Cube

§.§ Spectral Representation

In such representations, each pixel vector is isolated from other pixels and processed based on spectral signatures only which means the pixel is represented only in spectral space $x_i \in \mathcal{R}^{B}$. Where $B$ can either be the actual number of spectral channels or just relevant spectral bands extracted after some dimensionality reduction (DR) method. Usually, instead of using original spectral bands, a low dimensional representation of HSI is preferred for data processing in order to avoid redundancy and achieve better class separability, without considerable loss of useful information.

Dimensionality Reduction (DR) approaches for spectral HSI representation can either be supervised or unsupervised. Unsupervised techniques transform the high dimensional HSI into a low dimensional space without using the class label information, for example, Principal Component Analysis (PCA) and locally linear embedding [134]. On the other hand, supervised DR methods utilize labeled samples to learn the data distribution i.e. to keep data points of the same classes near to each other and separate the data points of different classes. For instance, linear discriminant analysis (LDA), local Fisher discriminant analysis (LFDA) [135], local discriminant embedding (LDE) [136] and nonparametric weighted feature extraction (NWFE) [137]. LDA and LDFA provide better class separability by maximizing the inter-class distance of data points and minimizing the intra-class distance. However, due to the spectral mixing effect, in which the same material may appear with different spectra or different materials may have the same spectral signatures, it becomes difficult to differentiate among different classes based on the spectral reflectance values alone.

§.§ Spatial Representation

To deal with the limitations of spectral representation, another approach is to exploit the spatial information of the pixels, in which pixels in each band are represented in the form of a matrix, $x_i \in \mathcal{R}^{N \times M}$. Due to high spatial correlation, neighboring pixels have higher probabilities to belong to the same class. Therefore, in the case of spatial representation, neighboring pixels’ information is also considered and the neighborhood of a pixel can be determined using kernel or pixel centric window [138]. Some common methods to extract spatial information from HSI cube are morphological profiles (MPs), texture features (like Gabor filters,  gray-level co-occurrence matrix (GLCM), and local binary pattern (LBP), etc.) and DNN based methods. Morphological profiles are capable of extracting geometrical characteristics. Few extensions of MPs include extended morphological profiles (EMPs) [139], multiple-structure-element morphological profiles [140], invariant attribute profiles (IAPs) [141].

The texture of the image provides useful spatial contextual information of HSI. For instance, a Gabor filter, a texture analysis technique, can efficiently obtain textural information at various scales and orientations. Similarly, LBP can provide rotation-invariant spatial texture representation. The GLCM can effectively determine the spatial variability of HSI by exploiting the relative positions of neighborhood pixels. The DNNs can also extract spatial information of HSI by considering the pixel as an image patch instead of representing it as a spectral vector. The spatial information contained in HSI can also be extracted by combining various of the afore discussed methods. For instance, [142]combined Gabor filter and differential morphological profiles [143] to extract local spatial sequential features for a recurrent neural network (RNN) based HSIC framework.

§.§ Spectral-Spatial Representation

This representation jointly exploits both spectral and spatial information of data. In such approaches, a pixel vector is processed based on spectral features while considering spatial-contextual information. The strategies that simultaneously use both spectral and spatial representations of HSI, either concatenate the spatial details with spectral vector [62, 144] or process the $3D$ HSI cube to preserve the actual structure and contextual information [145].

In literature, all these HSI representations are widely exploited for HSIC. Most of the DNNs for pixel-wise classification utilized the spectral representation of HSIs [146, 147]. However, to mitigate the limitations of spectral representation, many efforts have been made to incorporate the spatial information [148, 149]. Recently, joint exploitation of both spectral and spatial features has gained much popularity and led to improved classification accuracy [150, 151, 152, 153, 67, 154]. These HSI feature exploitation approaches, for HSIC, are further discussed in the following sections.

§ LEARNING STRATEGIES

Deep learning models can adopt various learning strategies that can be broadly categorized into the following:

§.§ Supervised Learning

In a supervised learning approach, the model is trained based on the labeled training data which means training data is comprised of a set of inputs and their corresponding outputs or class labels. During the training phase, the model iteratively updates its parameters in order to predict the desired outputs accurately. In the testing phase, the model is tested against the new input/test data in order to validate its ability to predict the correct labels. If trained sufficiently, the model can predict the labels of new input data. However, supervised learning of DNNs requires a lot of labeled training data to fine-tune the model parameter. Therefore, they are best suited to scenarios where plentiful labeled data is available. The details of various supervised learning techniques for DNNs will be explained in the respective sections.

§.§ Unsupervised Learning

In contrast to the supervised learning approach, unsupervised learning techniques learn from the input data with no explicit labels associated with it. These approaches try to identify the underlying statistical structure of input representations or patterns in the absence of corresponding labels. As there is no ground truth available for the training data so it might be difficult to measure the accuracy of the trained model. However, such learning strategies are useful in the cases where we want to learn the inherent structure of such datasets which have a scarcity of training data. The principal component analysis (PCA) is an unsupervised learning technique that can be used to learn a low-dimensional representation of the input. Similarly, k-means clustering is another unsupervised learning method that groups the input data into homogeneous clusters.

§.§ Semi-supervised Learning

The semi-supervised learning technique is halfway between unsupervised and supervised approaches. It learns from the partially labeled datasets that are a small amount of labeled training data can be utilized to label the rest of the unlabeled data. These techniques effectively utilize all available data instead of just labeled data, therefore, these techniques have gained much popularity among the research community and are being widely used for HSIC [155, 156, 157, 158]. The details of these methods are briefly described in section <ref>.

§ DEVELOPMENT OF DNNS (TYPES OF LAYERS)

In the following, we review recent developments of some widely used DNN frameworks for HSIC. We specifically surveyed the literature published from 2017 onward. DNNs exhibit a great variety of flexible and configurable models for HSIC that allow the incorporation of several types of layers. Few widely used types of layers are explained in the following.

A layer is the key building block of DNN and the type of layer has a decisive impact in terms of feature processing. A layer takes the weighted input, processes it through linear or non-linear transformation, and outputs these values to the next layer. Generally, a layer is uniform, as it has a single activation function. The first layer of the network is known as the input layer and the last layer as an output layer. All other layers in the network, in between the input and output layers, are known as hidden layers. These layers progressively find different features in the input data by performing various transformations. The choice of layer type depends on the task at hand, as some layers perform better for some tasks than others. The most commonly used layers for HSIC are explained below.

§.§ Fully Connected Layers

A fully connected (FC) layer connects every neuron in the lower layer to every neuron in the upper/next layer. Mostly, they are used as the last few layers of a model usually after convolution/pooling layers. FC takes the output of the previous layer and assigns weights to predict the probabilities for class labels. Due to a large number of connections, a large number of parameters need to be adjusted which significantly increases the computational overhead. Moreover, due to a large number of parameters, the model becomes more sensitive to overfitting [49]. However, to mitigate the effect of overfitting, a dropout method is introduced in [159].

§.§ Convolutional Layers

The convolutional (CONV) layer convolve the input data or feature maps from a lower layer with the filters (kernels). The filter contains weights whose dot product is calculated with the subset of input data by moving it across the width, height, and depth of the input region. The output of the filter is known as a feature map. CONV layer provides spatial invariance via a local connectivity approach in which the neuron in the feature map connects to a subset of input from the previous layer rather than connecting to every neuron. This reduces the number of parameters that need to train. To further reduce the number of parameters, the CONV layer uses the mechanism of parameter sharing in which the same weights are used in a particular feature map.

§.§ Activation Layers

Activation layers are assumed to be a feature detector stage of DNNs [160]. FC and CONV layers provide linear representations of input data or it can be said that they work similarly to linear regressors and data transformed by these layers is considered to be at the feature extraction stage [69]. Therefore, to learn non-linear features of data, an activation layer must be used after FC and CONV layers. In the activation layer, feature maps from previous layers go through an activation function to form an activation map. Some commonly used activation functions are sigmoid, hyperbolic tangent (tanh), rectified linear unit (ReLU), LiSHT [161] and softmax. However, in HSI analysis, softmax and ReLU are widely employed activation functions [69]. Figure <ref> presents a graphical representation of a few commonly utilized activation functions.

Graphical representation of various commonly used activation functions

§.§ Pooling or sub-sampling layers

The pooling layer, also known as the sub-sampling or down-sampling layer, takes a certain input volume and reduces it to a single value as shown in Figure <ref>. This provides invariance to small distortions in the data. The pooling layer helps the model to control overfitting as the size of data and model parameters both are reduced which also leads to a decrease in the computational time. The commonly used down-sampling operations are max-pooling, average-pooling, and sum-pooling. Recently, a pooling technique, wavelet-pooling is introduced in [162] whose performance is commensurable to max-pooling and average-pooling. Alternatively, [163] proposed another trend in which the pooling layer is replaced by the CONV layer of increased filter stride.

Max-pooling and average-pooling operations of down-sampling/pooling layer

§ CONVOLUTIONAL NEURAL NETWORK (CNN)

The architecture of the Convolutional Neural Network (CNN) is inspired by the biological visual system presented in [164]. Following the natural visual recognition mechanism proposed by Hubel and Wiesel [164], Neocognitron [165] is regarded as the first hierarchical, position-invariant model for pattern recognition [166] which can be considered as the predecessor of CNN [167]. The architecture of CNN can be divided into two main stages: one is Feature Extraction (FE) network and the other is a classification based on the feature maps extracted in the first stage.

The FE network consists of multiple hierarchically stacked CONV, activation, and pooling layers. The CONV layer extracts the features from input data by convolving a learned kernel with it. On each CONV layer, the kernel is spatially shared with whole input data which reduces the model’s complexity and the network becomes easier to train as the number of parameters that need to be fine-tuned is reduced. Convolved results are then passed through an activation layer which adds nonlinearities in the network to extract non-linear features of the input. This is achieved by applying a non-linear function to the convolved results. Afterward, the resolution of the feature map is reduced by applying a pooling operation to achieve shift-invariance. Generally, the pooling layer is added with every CONV layer followed by the activation function.

The classification stage consisting of FC layers and a Softmax operator gives the probability of input pattern belonging to a specific class based on the feature maps extracted at the FE stage. FC layer connects every single neuron in the previous layer to every neuron in the current layer. In [168] and [169], the authors proposed that the FC layer can be disregarded by using a global average pooling layer. Softmax is commonly used for classification tasks [170, 171] however, many works have also utilized SVM [172, 173] for this purpose.

In the following, we reviewed three types of CNN architectures for HSIC: i) Spectral CNN, ii) Spatial CNN and iii) Spectral-spatial CNN. Figure <ref> illustrates the general architecture of these three frameworks.

General architecture of Spectral CNN, Spatial CNN and Spectral-spatial CNN frameworks for HSIC.

§.§ Spectral CNN Frameworks for HSIC

Spectral CNN models only consider 1D spectral information $(x_i \in \mathcal{R}^{B})$ as input, where $B$ could either be the original number of spectral bands or the appropriate number of bands extracted after some dimensionality reduction method. In [174], a CNN structure was proposed to mitigate the overfitting problem and achieved a better generalization capability by utilizing $1 \times 1$ convolutional kernels and enhanced dropout rates. Moreover, a global average pooling layer is used in place of a fully connected layer in order to reduce the network parameters. To reduce high correlation among HSI bands [169] proposed a CNN architecture for HSIC which fully utilized the spectral information by transforming the 1D spectral vector to a 2D feature matrix and by cascading composite layers consisting of $1 \times 1$ and $3 \times 3$ CONV layers, the architecture achieved the feature reuse capability. Similar to [174], [169] also utilized the global average pooling layer to lower the network's training parameters and to extract high dimensional features.

In [175] authors presented a hybrid model for HSIC in which the first few CONV layers are employed to extract position invariant middle-level features and then recurrent layers are used to extract spectral-contextual details. Similarly, <cit.> used a hybrid architecture for classifying healthy and diseased Wheat heads. For the input layer, they transform spectral information into a 2D data structure. In [176] CNN proved to be more effective as compared to SVM and KNN for the spectral-based identification of rice seed’s variety. A similar application of CNN was explored in [147] where various varieties of Chrysanthemum were identified using spectral data of the first five PCs of Principal component analysis (PCA). PCA is a dimensionality reduction method that is widely used in many DL applications to handle/preprocess high dimensional data. In [177] PCA was utilized to preprocess medical HSI and then the fusion of CNN kernels with Gabor kernels using dot product is used for classification.

The study [178] analyzed another dimensionality reduction technique Dynamic Mode Decomposition (DMD) which converted 3D HSI data to 2D and then this data is fed to vectorized CNN (VCNN) for classification. To overcome the noise effect in pixel-wise HSIC, a method of averaged spectra is used in [179] where an averaged spectra of a group of pixels belonging to bacterial colonies is extracted for further analysis.

§.§ Spatial CNN frameworks for HSIC

Spatial CNN models only consider spatial information and to extract the spatial information from HSI data, dimensionality reduction (DR) methods are employed on spectral-domain to lower the dimensionality of original HSI data. For instance, [180] used PCA to extract the first PC with refined spatial information and fed it to a fully CNN framework for classification. Similarly, [181] trained a spatial-based 2D-CNN with one PC. In [182], PCA whitened input data considering three PCs is fed to a random patches network as a 2D-CNN classification framework. However, the limited training samples with highly similar spectral feature make DL models prone to over-fitting. To overcome this [183] proposed a probabilistic neighbourhood pooling based attention network (PNPAN) for HSI classification.

The method proposed in [184] cropped the patches from 2D input images (i.e. images from the different spectral bands) to train a 2D-CNN architecture that learns the data-adaptive kernels by itself. Furthermore, some authors also proposed the utilization of handcrafted features along with spectral-domain reduction. For example, [185] combined the Gabor filtering technique with 2D-CNN for HSIC to overcome the overfitting problem due to limited training samples. The Gabor filtering extracts the spatial details including edges and textures which effectively reduce the overfitting problem. The work [186] proposed a deformable HSIC network based on the concept of deformable sampling locations which can adaptively adjust their size and shape in accordance with HSI's spatial features. Such sampling locations are created by calculating 2D offsets for every pixel in the input image through regular convolutions by taking into account three PCs. These offsets can cover the locations of similar neighboring pixels possessing similar characteristics. Then structural information of neighboring pixels is fused to make deformable feature images. Regular convolution employed on these deformable feature images can extract more effective complex structures.

§.§ Spectral-Spatial CNN frameworks for HSIC

Spectral-spatial pixel-wise HSIC can be achieved by integrating spatial features into spectral information. For instance, [187] presented an improved pixel pair feature (PPF) approach called spatial pixel pair feature which is different from traditional PPFs with respect to two main aspects: one is the selection of pixel pair that is only the pixel from the immediate neighborhood of central pixel can be used to make a pair, second is the label of pixel pair would be as of central pixel. To extract discriminative joint representation [188] introduced a Supervised Spectral-Spatial Residual Network (SSRN) that uses a series of 3D convolutions in the respective spectral and spatial residual blocks. An efficient deep 3D-CNN framework was proposed in [189] that simultaneously exploits both spectral and spatial information for HSIC.

Similarly, to reflect the variations of spatial contexture in various hyperspectral patches, [190] implemented an adaptive weight learning technique instead of assigning fixed weights to incorporate spatial details. Besides this, to make the convolutional kernel more flexible [154] explored a new architectural design that can adaptively find adjustable receptive filed and then an improved spectral-spatial residual network for joint feature extraction. The discriminative power of the extracted features can be further improved by combining both the max and min convolutional features before the ReLU non-linearity reported in [191] for the classification task. CNN's are failed to exploit rotation equivariance in a natural way [192] introduced the translation equivariant representations of input features which provides extra robustness to the spatial feature locations for HSIC.

The deeper networks may suffer from the issues of overfitting and gradient vanishing problems due to the smaller number of available labeled training samples and to overcome this shortcoming the lightweight CNN's gain good attention in HSIC communities. The paper [193] introduced an end-to-end 3D lightweight convolutional neural network to tackle the limited numbers of training samples for HSI classification. To reduce the large gap between the massive trainable parameters and the limited labeled samples [194] proposed to extract the spatial-spectral Schroedinger eigenmaps (SSSE) joint spatial-spectral information, and then further reduced the dimensionality using compression technique. Approximately 90% of trainable weights of the total parameters are used immediately after the flatten operation i.e., in the fully connected layer, whereas the remaining only 10% weights are used on the previous convolutional layers of the whole network. To overcome the paper [195] introduced a lightweight bag-of-feature learning paradigm into an end-to-end spectral-spatial squeeze-and-excitation residual network for HSIC.

The morphological operations i.e., erosion and dilation are powerful nonlinear feature transformations that are widely used to preserve the essential characteristics of shape and structural information of an image. Inspired by these the paper [196] introduced a new end-to-end morphological convolutional neural network (MorphCNN) for HSIC which utilizes both the spectral and spatial features by concatenating the outputs from spectral and spatial morphological blocks extracted in a dual-path fashion.

The work [190] proposed a two-stage framework for joint spectral-spatial HSIC which can directly extract both spectral and spatial features instead of independently concatenating them. The first stage of the proposed network is comprised of a CNN and softmax normalization that adaptively learns the weights for input patches and extracts joint shallow features. These shallow features are then fed to a network of Stacked Autoencoder (SAE) to obtain deep hierarchical features and final classification is performed with a Multinomial Logistic Regression (MLR) layer. A 3D-CNN model was introduced in [197] to jointly exploit spectral-spatial features from HSI and to validate its performance comparison is performed with spectral-based DBN, SAE, and 2D-spatial CNN for HSIC. The work [198] introduced a bilinear fusion mechanism over the two branches of squeeze operation based on the global and max-pooling whereas the excitation operation is performed with the fused output of squeeze operation.

The work [199] proposed a deep multiscale spectral-spatial feature extraction approach for HSIC which can learn effective discriminant features from the images with high spatial diversity. The framework utilizes the Fully Convolutional Network (FCN) to extract deep spatial information and then, these features are fused with spectral information by using a weighted fusion strategy. Finally, pixel-wise classification is performed on these fused features.

In [200] a dual-channel CNN framework was implemented for spectral-spatial HSIC. In the proposed approach, 1D-CNN is used to hierarchically extract spectral features and 2D-CNN to extract hierarchical spatial features. These features are then combined together for the final classification task. Furthermore, to overcome the deficiency of training data and to achieve higher classification accuracy, the proposed framework is supported by a data augmentation technique that can increase the training samples by a factor of 6. In [201], a multiscale 3D deep CNN is introduced for end-to-end HSIC which can jointly learn both 1D spectral and 2D multiscale spatial features without any pre-processing or post-processing techniques like PCA, etc. In order to reduce the band redundancy or noise in HSI, [202] explored a novel architecture for HSIC by embedding a band attention module in the traditional CNN framework. The study [203] proposed an HSIC architecture in which PCA transformed images are used to obtain multi-scale cubes for handcrafted feature extraction by utilizing multi-scale covariance maps which can simultaneously exploit spectral-spatial details of HSI. These maps are then used to train the traditional CNN model for classification.

The work [204] combined CNN with metric learning-based HSIC framework which first utilizes CNN to extract deep spatial information using the first three PCs extracted by PCA. Then, in a metric learning-based framework, spectral and spatial features are fused for spectral-spatial feature learning by embedding a metric learning regularization factor for the classifier’s training (SVM). Similarly, [205] combines multi-scale convolution-based CNN (MS-CNN) with diversified deep metrics based on determinantal point process (DPP) [206] priors for (1D spectral, 2D spectral-spatial, and 3D spectral-spatial) HSIC. Multiscale filters are used in CNN to obtain multi-scale features and DPP-based diversified metric transformation is performed to increase the inter-class variance and decrease intra-class variance, and better HSI representational ability. Final classification maps are obtained by using a softmax classifier.

In recent work, [207] an HSIC framework is proposed to extract multi-scale spatial features by constructing a three-channel virtual RGB image from HSI instead of extracting the first three PCs through PCA. The purpose of using a three-channel RGB image is to utilize existing networks trained on natural images to extract spatial features. For multi-scale feature extraction, these images are passed to a fully convolutional network. These multi-scale spatial features are fused and further joined with PCS extracted spectral features for final classification via SVM.

A two-branch (spectral and spatial) DNN for HSIC was introduced in [208]. The spatial branch consists of a band selection layer and a convolutional and de-convolutional framework with skip architecture to extract spatial information of HSI, and in the spectral branch, a contextual DNN is used to extract spectral features. The paper [209] introduced an adaptive band selection based semi-supervised 3D-CNN to jointly exploit spectral-spatial features whereas [210] explored dual-attention based autoencoder-decoder network for unsupervised hyperspectral band selection and then joint feature extraction for land cover class prediction. Similarly, in [211] spectral-spatial features are simultaneously exploited in an unsupervised manner using a 3D convolution autoencoder. The pixel-wise land use and land cover (LULC) classification using traditional CNNs is often suffered by the presence of wrong / noisy labels in the training set and can easily be overfitted to the labeled noises. To overcome this problem of accurate classification [212] proposed a lightweight heterogeneous kernel convolution (HetConv3D) for HSI classification with noisy labels by effectively combining both the spectral and spatial kernel feature to produce discriminative and invariant feature maps for classification.

A hybrid 3D-2D-CNN architecture was presented by [213] in which 3D-CNN is first used to extract joint spectral-spatial features and then 2D-CNN is further used to obtain more abstract spatial contextual features. The study [214] proposed to use adaptive Markov random field for HSIC. The CNN first extracts joint spectral-spatial features and then a smooth MRF prior is placed on class labels to further refine the spatial details. Convolutional neural networks are greatly affected by overfitting and vanishing gradient problems and to overcome this a separable attention network was introduced by [215]. Where the input feature maps are divided into several groups and split along the channel dimension and finally an attention mask encodes global contextual information by combining them. Recently, generalized gradient centralized $3D$ convolution (G2C-Conv3D) was introduced in [216] to combine both the intensity level semantic information and gradient level detailed information extracted from raw HSIs during the convolutions operation. To boost the performance of accurate land-cover types classification, G2C-Conv3D can be easily plugged into the existing HSIs feature extraction networks.

§.§ GCN frameworks for HSIC

Graph Convolutional Networks (GCNs) [217] have been garnering increasing attention to researchers in various application fields, owing to their flexible and diversified network architecture that is capable of processing non-grid high-dimensional data. Such properties provide new insight and possibilities in processing hyperspectral data more effectively and efficiently. In detail, GCNs enable the modeling of the relations between data (or samples). Accordingly, this naturally motivates us to use the GCNs to capture the spatial relations of spectral signatures in HSIs. Due to the GCNs' limitations in the graph construction [218], particularly for large graphs (need expensive computational cost), GCNs fail to classify or identify materials in large-scale hyperspectral scenes using normal PCs, which leads to relatively less popularity compared to CNN’s in HSIC. For this reason, there have been some tentative researches using the GCNs in the HSIC task.

For example, a second-order GCN was proposed in [219] by modeling spatial-spectral relations on manifolds for HSIC by the attempts to reduce the computational cost on graphs. Authors of [220] first used superpixel segmentation techniques on HSIs and fed superpixels instead of pixels into GCNs. This enables the network training of GCNs on a large number of pixels in HSIs with the application to the land cover classification task. Nevertheless, these methods still fail to solve the problem of GCNs essentially. To this end, Hong et al. [218] proposed a novel miniGCN. As the name suggests, miniGCN trains the GCNs in a mini-batch fashion, which is the same as CNN. The proposed miniGCN not only reduces the computational cost-effectively but also makes it possible to make a quantitative comparison and fusion with CNNs, further yielding a FuNet for HSIC.

§.§ Future directions for CNN-based HSIC

In the preceding section, we have reviewed the recent developments of CNNs for HSIC. Although CNN's based HSIC frameworks have achieved great success with respect to classification performance, there are still many aspects that need further investigation. For instance, there is a need to further work on such models that can jointly employ spatial and spectral information for HSIC. Many of the above-surveyed frameworks use dimensionality reduction methods to achieve better spectral-spatial representation but such approaches discard useful spectral information of HSI. Hence the development of robust HSIC approaches that can preserve spectral information is required. However, the processing of such approaches increases the computational burden, and the training process becomes slower, therefore, parallel processing of such networks using FPGAs and GPUs is desired in order to achieve the computationally fast models, that can even be suitable for mobile platforms, without the performance degradation.

Moreover, as the CNNs are becoming deeper and deeper, more labeled training data is required for accurate classification, and as discussed before, there is a lack of labeled training data in HSI. In order to overcome this issue, more research is required to integrate the CNN with unsupervised or semi-supervised approaches. Furthermore, we should pay more attention to the generalization ability of CNNs, particularly for the input data format (not only limiting to the grid data). GCNs might be a good solution to combine with CNN's together to develop a more general CNN-based new framework. Using this, we expect to be able to further break the performance bottleneck, yielding more efficient HSIC.

§ AUTOENCODERS (AE)

Autoencoder (AE) is a popular symmetrical neural network for HSIC due to its unsupervised feature learning capability. AE itself does not perform a classification task instead it gives a compressed feature representation of high-dimensional HSI data. AE consists of an input layer, one hidden or encoding layer, one reconstruction or decoding layer, and an output layer as shown in Figure <ref>. AE is trained on input data in such a manner to encode it into a latent representation that is able to reconstruct the input. To learn a compressed feature representation of input data, AE tries to reduce the reconstruction error that is minimizing the difference between the input and the output.

A general Autoencoder Architecture

Whereas, the Stacked Autoencoder (SAE) is built by stacking multiple layers of AEs in such a way that the output of one layer is served as an input of the subsequent layer. Denoising autoencoder (DAE) is a variant of AE that has a similar structure as AE except for the input data. In DAE, the input is corrupted by adding noise to it, however, the output is the original input signal without noise. Therefore, DAE, different from AE, can recover original input from a noisy input signal.

To learn high-level representation from data, the work [221] proposed a combination of multi-layer AEs with maximum noise fraction which reduces the spectral dimensionality of HSI, while a softmax logistic regression classifier is employed for HSIC. The study reported in [222] combined multi-manifold learning framework proposed by [223] with Counteractive Autoencoder [224] for improved unsupervised HSIC. The work [225] jointly exploited spectral-spatial features of HSI through an unsupervised feature extracting framework composed of recursive autoencoders (RAE) network. It extracts the features from the neighborhood of the target pixel and weights are assigned based on the spectral similarity between target and neighboring pixels. A two-stream DNN with a class-specific fusion scheme was introduced in [226] which learns the fusion weights adaptively. One stream composed of stacked denoising auto-encoder is used to extract spectral features and the second stream is implemented to extract spatial information using Convolutional Neural Network (CNN), while final classification is performed by fusing the class prediction scores obtained from the classification results of both streams.

Another work proposed a hybrid architecture for multi-feature based spectral-spatial HSIC which utilizes PCA for dimensionality reduction, guided filters [227] to obtain spatial information, and sparse AE for high-level feature extraction. The framework proposed in [228] exploited both spectral and spatial information for HSIC by adopting batch-based training of AEs and features are generated by fusing spectral and spatial information via a mean pooling scheme. Another work [229] developed a spectral-spatial HSIC framework by extracting appropriate spatial resolution of HSI and utilization of stacked sparse AE for high-level feature extraction followed by Random Forest (RF) for the final classification task.

Similarly, [230] also used stacked sparse AE for various types of representation that is spectral-spatial and multi-fractal features along with other higher-order statistical representations. A combination of SAE and extreme learning machine was proposed in [231] for HSIC, which segments the features of the training set and transform them via SAE, after transformation, feature subsets are rearranged according to the original order of the training set and fed to extreme learning machine-based classifiers, while Q-statistics is used for final classification result. This processing of feature subsets helps to improve variance among base classifiers [231]. Similarly, in a recent work [232] implemented a computationally efficient multi-layer extreme learning machine-based AE which learns the features in three folds, as proposed in [39] for HSIC.

To overcome the issue of high intra-class variability and high inter-class similarity in HSI, [233] developed an SAE-based HSIC which can learn compact and discriminative features by imposing a local fisher discriminant regularization. Similarly, in the latest work [234] a k-sparse denoising AE is spliced with and spectral–restricted spatial features that overcome the high intra-class variability of spatial features for HSIC. The study [235] proposed an HSIC architecture that first makes the spectral segments of HSI based on mutual information measure to reduce the computation time during feature extraction via SAE, while spatial information is incorporated by using extended morphological profiles (EMPs) and SVM/RF is used for final classification. Recently, [236] used SAE for the classification of an oil slick on the sea surface by jointly exploiting spectral-spatial features of HSI.

§.§ Future Directions for AE-based HSIC

In the above section, we have surveyed the recent developments of AEs based techniques for HSIC. Although such frameworks provide powerful predictive performance and show good generalization capabilities, more sophisticated work is still desired. Many of the discussed approaches do not fully exploit abundant spatial information so further techniques need to be developed that can fully employ joint spatial and spectral information for HSIC. Moreover, the issue of high intra-class variability and high inter-class similarity in HSI also hinders the classification performance. Many of the above-reviewed works have addressed this issue but further research to overcome this aforesaid issue is required. One direction could be further exploring approaches like pre-training, co-training, and adaptive neural networks, etc for AE-based HSIC frameworks.

§ DEEP BELIEF NETWORK (DBN)

Deep Belief Network (DBN) [237] is a hierarchical deep DNN that learns the features from input in an unsupervised, layer-by-layer approach. The layers in DBN are built using Restricted Boltzmann Machine (RBM) comprised of a two-layer architecture in which visible units are connected to hidden units [238] as shown in Figure <ref>.

Basic architecture of RBM

A detailed overview of RBM can be found at [238]. To extract more comprehensive features from input data, the hidden unit of one RBM can be fed to the visible units of other RBM. This type of layer-by-layer architecture builds a DBN, which is trained greedily and can capture deep features from HSI. The architecture of three-layer DBN is shown in Figure <ref>.

A three layer DBN architecture

In literature, several works implemented DBN for HSIC. For instance, [239] used DBN for land cover classification by combining spectral-spatial information and making a comparison with some other classification approaches. The usual learning process of DBN involves two steps: one is unsupervised pre-training with unlabeled samples and the second is supervised fine-tuning with the help of labeled samples. However, this training process may result in two problems: first, multiple hidden units may tend to respond similarly [240] due to co-adaptation [241] and second is linked with the sparsity and selectivity of activations neurons that are some neurons may always be dead or always responding [242]. To mitigate these two problems, [243] introduced a diversified DBN model through regularizing the pre-training and fine-tuning process by imposing a diversity prior to enhancing the DBN's classification accuracy for HSI.

To extract efficient texture features for the HSIC, the work [244] proposed a DBN based texture feature enhancement framework that combines band grouping and sample band selection approach with a guided filter to enhance the texture features, which are then learned by a DBN model and final classification results are obtained by a softmax classifier. The work [245] implemented a parallel layers framework consisting of Gaussian-Bernoulli RBM which extracts high-level, local invariant, and nonlinear features from HSI and a logistic regression layer is used for classification.

To improve the classification accuracy, some works are considered to jointly exploit the spectral and spatial information contained in HSI. For instance, [246] introduced a DBN framework with the logistics regression layer and verified that the joint exploitation of spectral-spatial features leads to improved classification accuracy. Similarly, [247] proposed a spectral-spatial graph-based RBM method for HSIC which constructs the spectral-spatial graph through joint similarity measurement based on spectral and spatial details, then an RBM is trained to extract useful joint spectral-spatial features from HSI, and finally, these features are passed to a DBN and logistic regression layer for classification.

§.§ Future directions for DBN-based HSIC

In the preceding section, we have reviewed the latest developments of DBN-based HSIC frameworks. We have observed that relative to other DNNs, very few works have utilized the DBNs for HSIC. Therefore, there is a need to further explore the DBN-based robust techniques that can jointly employ spatial and spectral features for HSIC. In addition, another research direction can be the regularization of the pretraining and fine-tuning processes of DBN to efficiently overcome the issue of dead or potentially over-tolerant (always responding) neurons.

§ RECURRENT NEURAL NETWORK (RNN)

The architecture of the Recurrent Neural Network (RNN), shown in Figure <ref>, comprises loop connections, where the node activation of the next step depends on the previous step [248]. Therefore, RNNs are capable of learning temporal sequences. RNN models process the spectral information of HSI data as time sequence considering the spectral bands as time steps [249]. There are three basic models of RNN a) Vanilla, b) Long-Short-Term Memory (LSTM) and c) Gated Recurrent Unit (GRU).

RNN architecture

Vanilla is the simplest RNN model and leads to information degradation while processing high-dimensional data. LSTM models composed of two states overcome this issue by controlling the information flow through three gates: input, forget, and output gates. It learns the relevant information over time by discarding the extraneous information. However, the gate controlling strategy makes the LSTM a considerably complex approach. GRU variant of LSTM enjoys the simplicity of the Vanilla model and provides high performance similar to LSTM. GRU is a simpler version of LSTM which modifies the input and forget gate as an update ($z_t$) and reset ($r_t$) gate and removes the output gate. A comparison of LSTM and GRU's internal architecture is presented in Figure <ref>.

Internal architecture of LSTM and GRU

The work [70] proposed an RNN based HSIC framework with a novel activation function (parametric rectified tanh) and GRU, which utilizes the sequential property of HSI to determine the class labels. In [142] a local spatial sequential (LSS) method based RNN framework was introduced which first extracts low-level features from HSI by using Gabor filter and differential morphological profiles [143]  and then fuse these features to obtain LSS features from the proposed method, these LSS features are further passed to an RNN model to extract high-level features, while a softmax layer is used for final classification.

Keeping in view the usefulness of spatial information to achieve improved classification accuracies, the work [250] proposed a spectral-spatial LSTM based network that learns spectral and spatial features of HSI by utilizing two separate LSTM followed softmax layer for classification, while a decision fusion strategy is implemented to get joint spectral-spatial classification results. Similarly, [251] proposed a patch-based RNN with LSTM cells that incorporate multi-temporal and multi-spectral information along with spatial characteristics for land cover classification.

In literature, several works proposed CNN-based hybrid RNN architectures (CRNN) for HSIC. For instance, [175] implemented a convolutional RNN in which the first few CONV layers are employed to extract position invariant middle-level features, and then recurrent layers are used to extract spectral-contextual details for HSIC. Similarly, [252] utilized such a model for semi-supervised HSIC by using pseudo labels. The study [253] suggested an HSIC framework in which CNN is used to extract spatial features from HSI, then these features are passed to a GRU-based fusion network that performs feature level and decision level fusion.

Similarly, Luo, et al., [254] exploited both spectral and spatial information contained in HSI by combining CNN with parallel GRU-based RNN which simplifies the training of GRU and improves performance. Bidirectional Convolutional LSTM (CLSTM) was proposed in [153] to jointly exploit spectral-spatial feature of HSI for classification. In, [255] combined multiscale local spectral-spatial features extracted by 3D-CNN with a hierarchical RNN which learns the spatial dependencies of local spectral-spatial features at multiple scales. Recurrent 2D-CNN and recurrent 3D-CNN for HSIC were proposed in [256] and along with an interesting comparison of these frameworks with their corresponding 2D and 3D-CNN models, which validates the superiority of recurrent CNN. The work [257] integrated CNN with CLSTM in which a 3D-CNN model is used to capture low-level spectral-spatial features and CLSTM recurrently analyzes this low-level spectral-spatial information. Recently, [70], introduced a cascade RNN for HSIC which consist of two layers of GRU-based RNN, the first layer is used to reduce the redundant spectral bands and the second layer is used to learn the features from HSI, furthermore, a few convolutional layers are employed to incorporate the rich spatial information contained in HSI.

§.§ Future directions for RNN-based HSIC

In the above section, we have surveyed the recent developments of AEs based techniques for HSIC. Although RNN-based HSIC frameworks have attracted considerable attention to the remote sensing community and achieved great success for classification performance, there are still many aspects that need further investigation. For instance, the construction of sequential input data for RNN. Most of the surveyed methods considered HSI pixel as a sequential point that is the pixel from each spectral band that forms a data sequence. However, This increases the length of RNN’s input sequence considerably large which can lead to an overfitting issue.

Moreover, processing such large data sequences increases the computational time and the learning process becomes slower. Therefore, the use of parallel processing tools needs to be further investigated to achieve good generalization performance of RNN-based HSIC. In addition, approaches like a grouping of spectral bands to decrease the data sequence length and utilization of the entire spectral signature to better discriminate between various classes can further be explored to construct the sequential input of the RNN model. Another interesting future direction may involve the implementation of RNN-based HSIC frameworks in a real multi-temporal HSI context.

§ STRATEGIES FOR LIMITED LABELED SAMPLES

Although DNNs have been successfully exploited for the task of HSIC however, they require a considerably large amount of labeled training data. However, as discussed earlier, the collection of labeled HSI is very critical and expensive due to numerous factors that either demand human experts or exploration of real-time scenarios. The limited availability of labeled training data hinders classification performance. To overcome the aforesaid issue, many effective strategies have been proposed in the literature. In this section, we will briefly discuss some of these strategies while focusing on active learning algorithms.

§.§ Data Augmentation

To combat the issue of limited training samples, data augmentation is proven to be an effective tool for HSIC. It generates new samples from the original training samples without introducing additional labeling costs. Data augmentation approaches can be categorized into two main strategies as i) data wrapping; ii) oversampling [258]. Data wrapping usually encodes several invariances (translational, size, viewpoint, and/or illumination) by conducting geometric and color-based transformations while preserving the labels, and oversampling-based augmentation methods inflate the training data by generating synthetic samples based on original data distributions. Oversampling techniques include mixture-based instance generation, feature space augmentations [258], and Generative Adversarial Networks (GANs) [259].

Referring to HSIC literature, several data augmentation-based frameworks have been employed to improve the classification performance by avoiding potential overfitting, which is generally caused by the limited availability of training data. For instance, [260] enhanced the training data by using three data augmentation operations (flip, rotate, and translation), and then this enhanced data is exploited to train CNN for HSIC.  The work [261] presented a comprehensive comparison of various extensively utilized HSI data augmentation techniques and proposed a pixel-block pair-based data augmentation that utilized both spectral and spatial information of HSI to synthesis new instances, to train a CNN model for HSIC. The work [262] compared the classification performance of a combination of CNN and AL with and without data augmentation techniques and demonstrated that the data augmentation leads to higher classification accuracies. Similarly, in another comparison [263], data augmentation-based CNN exhibited a 10% increase in HSIC accuracy when compared to a PCA-based CNN model.

The above-discussed methods utilize offline data augmentation techniques that increase the training data by creating new instances during/before the training process of a model. Recently, a novel data augmentation framework for HSI is proposed in [264] which, rather than inflating the training data, generates the samples at test time, and a DNN trained over original training data along with a voting scheme is used for the final class label. To improve the generalization capability of DNN models, the work [264] also proposed two fast data augmentation techniques for high-quality data syncretization. A similar PCA-based online data augmentation strategy is proposed in [265] which also synthesis new instances during the inference, instead of training.

§.§ Semi-Supervised/Unsupervised Learning

Semi-Supervised Learning (SSL) approaches learn data distribution by jointly exploiting both labeled and unlabeled data. These techniques expand the training data by utilizing unlabeled samples along with labeled ones in order to construct a relationship between feature space and class labels. Several SSL-based HSIC frameworks have been proposed in the literature that can mainly be categorized as follows: i) Co-training, ii) Self-training, iii) GANs, iv) Graph-based SSL models and v) Semi-supervised SVM. A recent comprehensive survey on these SSL techniques can be found in [266]. Moreover, another in-depth survey of SSL approaches is also presented in [267].

The SSL-based HSIC techniques are briefly summarized in [268], where authors also made a detailed comparison of these methods. The method presented in [252] used pseudo or cluster-labeled samples to pre-train a CRNN for HSIC and small-sized labeled data is used to fine-tune the network. Similarly, [156] proposed a semi-supervised HSIC framework that exploits PCA and extended morphological attribute profiles to extract pseudo-labeled samples which are fed to a CNN-based deep feature fusion network.

The work [269] proposed a dual strategy co-training approach based on spectral and spatial features of HSI. Similarly, [270] separately pre-trained two SAEs, one using spectral and the other using spatial features of HSI, and fine-tuning is achieved via a co-training approach. [271] proposed a region information-based self-training approach to enhance the training data. A graph-based self-training framework was developed in [272] where initial sampling is achieved through subtractive clustering. Recently, [157] improved the HSIC performance by pseudo-labeling the unlabeled samples through a clustering-based self-training mechanism and regulating the self-training by employing spatial constraints.

§.§ Generative Adversarial Network (GAN)

GAN proposed by [273], is comprised of two neural networks, one is known as a generator and the other is known as discriminator (Figure <ref>). GANs can learn to replicate the samples by exploiting the data distribution details. The work [274] proposed a spectral feature-based GAN for SSL-based HSIC.

A general architecture of Generative Adversarial Network (GAN)

Similarly, [275] proposed a GAN-based spectral-spatial HSIC framework. Similarly, [276] developed CNN-based 1D-GAN and 3D-GAN architectures to enhance the classification performance. A 1D customized GAN is used to generate the spectral features [277], which is further used by CNN for feature extraction, and then majority voting is performed HSIC. Very recently, [278] introduced a spatial-spectral multi-class GAN (MSGAN) which utilizes two generators to produce spatial and spectral information with the help of multiple adversarial objectives. To address the data imbalance problem for HSI classification [279] proposed a new semi-supervised model which combines GAN with conditional random fields (CRFs).

Similarly,  [280] investigated a Caps-TripleGAN model which effectively generates new samples using a 1D structure Triple Generative Adversarial Network (TripleGAN) and classifying the generated HSI samples using the capsule network (CapsNet). The work [281] proposed to utilize a 3D CNN-based generator network and a 3D deep residual network-based discriminator network for HSIC. To learn high-level contextual features combination of both capsule network and convolutional long short-term memory (ConvLSTM) based discriminator model has been proposed in [282] for HSIC.

The work [283] proposed to address the scarcity of training examples by utilizing a GAN model where the performance of the discriminator is further improved by an auxiliary classifier to produce more structurally coherent virtual training samples. Besides this, to enhance the model performance [284] proposed a generative adversarial minority oversampling-based technique for addressing the long-standing problem of class-wise data imbalanced imposed by HSIC.

§.§ Transfer Learning

Transfer learning enhances the performance of a model by using prior knowledge of a relevant primary task to perform a secondary task. In other words, information extracted from the relevant source domain is transferred to the target domain to learn unseen/unlabeled data. Therefore, transfer learning can be effectively employed in domains with insufficient or no training data. Based on the availability of labeled training instances, transfer learning frameworks can further be categorized as supervised or unsupervised transfer learning. Generally, both source and target domains are assumed to be related but not exactly similar. However, they may follow different distributions as in the case of HSIC where categories of interest are the same but data in two domains may vary due to different acquisition circumstances.

In DNN based HSIC, the model learns features in a hierarchical manner, where lower layers usually extract generic features, when trained on various images. Therefore, the features learned by these layers can be transferred to learn a new classifier for the target dataset. For instance, [285] pertained to a two-branch spectral-spatial CNN model with an ample amount of training data from other HSIs and then applied the lower layers of the pre-trained model to the target network for the robust classification of target HSI. To learn the target-specific features, higher layers of the target network are randomly initialized and the whole network is fine-tuned by utilizing limited labeled instances of target HSI. Similarly, [286] proposed a suitable method to pre-train and fine-tune a CNN network to utilize it for the classification of new HSIs. The study [287] combined data augmentation and transfer learning approaches to combat the shortage of training data in order to improve HSIC performance.

As discussed before, data in source and target domain may vary in many aspects, for instance, in the case of HSIs, the dimensions of two HSIs may vary due to the acquisition from different sensors. Handling such cross-domain variations and transferring the knowledge between them is known as heterogeneous transfer learning (a detailed survey of such methods can be found in [288]). In HSIC literature, several works have been proposed to bridge the gap for transferring the knowledge between two HSIs, with varying dimensions and/or distributions.

For example, [289] proposed an effective heterogeneous transfer learning-based HSIC framework that works well with both homogeneous and heterogeneous HSIs, and [290] used an iterative re-weighting mechanism-based heterogeneous transfer learning for HSIC. Similarly, a recent work [291] proposed a band selection-based transfer learning approach to pre-train a CNN, which retains the same number of dimensions for various HSIs. Furthermore, [292] proposed an unsupervised transfer learning technique to classify completely unknown target HSI and [293] demonstrate that the networks trained on natural images can enhance the performance of transfer learning for remote sensing data classification as compared to the networks trained from scratch using smaller HSI data.

§.§ Active Learning

Active Learning (AL) iteratively enhances the predictive performance of a classifier by actively increasing the size of training data, for each training iteration, by utilizing an unlabeled pool of samples. In each iteration, AL enhances the training dataset by actively selecting the most valuable instances from the pool of unlabeled data and an oracle (Human or machine-based) assigns the true class labels to these instances. Finally, these useful instances are added to the existing training dataset and the classifier is retrained on this new training dataset. The process continues until a stopping criterion, that maybe the size of the training dataset, the number of iterations, or the desired accuracy score, is achieved. A general framework of AL is illustrated in Figure <ref>.

A general overview of Active Learning

The selection of the most useful/effective samples is made in such a way that the samples should be informative and representative of the overall input distribution in order to improve accuracy. Based on the criteria of adding new instances to the training set, AL frameworks can be designated as either stream-based or pool-based. In stream-based selection, one instance at a time is drawn from an actual set of unlabeled samples and the model decides whether to label it or not based on its usefulness. While in pool-based strategy, samples are queried from a pool/subset of unlabeled data based on ranking scores computed from various measures to evaluate the sample's usefulness.

The work [294] found that streamed-based selection gives poorer learning rates as compared to pool-based selection as the former tends to query extra instances. In pool-based selection, it is important to incorporate diversity in the pool of samples, in order to avoid redundancy within the pool of samples. Generally, the following three aspects are focused on while selecting/querying the most valuable samples: heterogeneity behavior, model’s performance, and representativeness of samples. A brief introduction of these sampling approaches is given below:

§.§.§ Heterogeneity-based selection

These approaches select the samples that are more heterogeneous to the already seen instances with respect to model diversity, classification uncertainty, and contention between a committee of various classifiers. Uncertainty sampling, expected model change, and query-by-committee are examples of heterogeneity-based models.

* Uncertainty Sampling: In this approach, the classifier iteratively tries to query the label of those samples for which it is most uncertain while predicting the label. The selection of new instances is based on ranking scores against a specified threshold and the instances with scores closest to that threshold are queried for labels.  One simple example of such a scheme could be implementing the probabilistic classifier on a sample in a scenario of binary classification and querying its label if the predicted class probability is close to $0.5$.

* Query-by-Committee: Such heterogeneity-based approaches perform the sampling process based on the dissimilarities in the predictions of various classifiers trained on the same set of labeled samples. A committee of various classifiers trained on the same set of training data is used to predict the class labels of unlabeled samples and the samples for which classifiers differ more are selected for querying labels. The committee of different classifiers can either be built by using ensemble learning algorithms like Bagging and Boosting [295] or by changing the model parameters [296]. Generally, a less number of diverse classifiers is adequate for constructing a committee [297, 295].

* Expected Model Change: Such a heterogeneity-based approach chooses the instances which result in a significant change from the current model in terms of the gradient of the objective function. Such techniques attempt to query the label for those instances that are considerably different from the current model. These sampling techniques only fit the models which follow gradient-based training procedures/optimization.

§.§.§ Performance-based Selection

Such methods consider the effect of adding queried samples to the model performance. They try to optimize the performance of the model by reducing variance and error. There are two types of performance-based sampling:

* Expected Error Reduction: This approach is interrelated to uncertainty sampling in such a way that uncertainty measures maximize the label uncertainty of the sample to be queried for the label while expected error reduction reduces the label uncertainty of the queried sample. Referring to the already discussed example of the binary classification problem, the expected error reduction approach would choose the samples with a probability far away from $0.5$ in order to reduce the error rate. Such techniques are also known as the greatest certainty models [296].

* Expected Variance Reduction: Reducing the variance of the model is guaranteed to reduce future generalization error [298]. Therefore, expected variance reduction techniques attempt to indirectly reduce the generalization error by minimizing the model variance. Such approaches query the instances that result in the lowest model variance. The Fisher information ratio is a well-known variance minimization framework.

§.§.§ Representativeness-based selection

Heterogeneity-based models are prone to include outlier and controversial samples but performance-based approaches implicitly avoid such samples by estimating future errors. Representative sampling tends to query such instances that are representative of the overall input distribution, hence, avoid outliers and unrepresentative samples. These approaches weigh the dense input region to a higher degree while the querying process. Density-weighted techniques like information density are examples of representativeness sampling approaches that consider the representativeness of samples along with heterogeneity behavior, and are also known as hybrid models [296].

Recently, AL has been intensively utilized in HSIC. [299] proposed a feature-driven AL framework to define a well-constructed feature space for HSIC. [300] proposed a Random Forest-based semi-supervised AL method that exploits spectral-spatial features to define a query function to select the most informative samples as target candidates for the training set.

Spatial information has been intensively exploited in many AL-based HSIC. For instance, [301] presented an AL framework that splice together the spectral and spatial features of superpixels. Similarly, [302] considered the neighborhood and superpixel information to enhance the uncertainty of queried samples. In recent work, [303] exploited the attribute profiles to incorporate spatial information in an AL-based HSIC framework.

Batch-mode AL frameworks have been widely employed to accelerate the learning process. Such approaches select a batch of samples, in each iteration, to be queried for a label. Therefore, the diversity of the samples is extremely critical in batch mode AL techniques in order to avoid redundancy. A multi-criteria batch-mode AL method proposed by [304] defines a novel query function based on diversity, uncertainty, and cluster assumption measures. These criteria are defined by exploiting the properties of KNN, SVM, and K-means clustering respectively, and finally, genetic algorithms are used to choose the batch of most effective samples. Similarly, [305] proposed a regularized multi-metric batch-mode AL framework for HSIC that exploits various features of HSI.

A multiview AL (MVAL) framework was proposed in [306] that analyzes the object from various views and measure the informativeness of the sample through multiview Intensity-based query criteria. Similarly, [307] also exploited the concept of multiview learning using the Fisher Discriminant Ratio to generate multiple views. In another work, [308] proposed a novel adaptive MVAL framework for HSIC which jointly exploits the spatial and spectral features in each view. Recently, [309] proposed an MVAL technique that utilizes pixel-level, subpixel-level, and superpixel-level details to generate multiple views for HSIC. Moreover, the proposed method exploits joint posterior probability estimation and dissimilarities among multiple views to query the representative samples.

In the HSIC literature, several works have combined the AL and DNN. For instance, [310] joined autoencoder with AL technique and [311] proposed a DBN-based AL framework for HSIC. Similarly, [312] coupled Bayesian CNN with AL paradigm for spectral-spatial HSIC. Recently, [262] proposed a CNN-based AL framework to better exploit the unlabeled samples for HSIC.

Many works integrated AL with transfer learning for HSIC. For example, [313] proposed an AL-based transfer learning framework that extracts the salient samples and exploits high-level features to correlate the source and target domain data. Another work, [314] proposed a Stacked Sparse AE-based Active Transfer Learning technique that jointly utilizes both spectral and spatial features for HSIC. Another work [315] combined domain adaptation and AL methods based on multiple kernels for HSIC.

AL-based HSIC offers some sophisticated frameworks to enhance the generalization capabilities of models. For instance, [35] proposed a fuzziness-based AL method to improve the generalization performance of discriminative and generative classifiers. The method computes the fuzziness-based distance of each instance and estimated class boundary, and the instances having greater fuzziness values and smaller distances from class boundaries are selected to be the candidates for the training set. Recently, [316] proposed a non-randomized spectral-spatial AL framework for multiclass HSIC that combines the spatial prior Fuzziness approach with Multinomial Logistic Regression via a Splitting and Augmented Lagrangian classifier. The authors also made a comprehensive comparison of the proposed framework with state-of-the-art sample selection methods along with diverse classifiers.

§ EXPERIMENTAL EVALUATION

The most research-oriented works published in the literature present a comprehensive experimental evaluation to highlight the pros and cons of the work/s proposed. However, to some extent, these works may have chosen different experimental settings, for instance, training, validation, and test samples may have the same number or percentage but the samples may be different as these samples are normally chosen randomly. Therefore, to make a fair comparison among different works proposed in the literature, one must need to have the same experimental settings.

These experimental settings include the same samples (geographical locations should remain the same for all chosen models no the different ones) and the number of samples should have been selected for each round of training in the cross-validation process. Normally, these samples have been chosen randomly, thus high likely, they may be different for different models if the models are executed at different times.

The other issue with most of the literature proposed in recent years is overlapping between training/test samples, i.e., training/validation samples have been randomly selected (including or excluding the above point) for training and validation however, the entire dataset has been passed at a testing phase which leads to a highly biased model (as the training samples have already been seen by the model) and produces high accuracy. Thus, in this work, the training/test samples are though chosen randomly (because all the models have been executed at the same time) however, the above point has been taken seriously and the intersection among these samples remain empty.

§.§ Experimental Datasets

The Indian Pines (IP) dataset was gathered by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) [317] over the Indian Pines test site in North-western Indiana. It contains $224$ spectral bands within a wavelength range of $400$ to $2500$ $nm$. The $24$ null and corrupted bands have been removed. The spatial size of the image is $145\times{145}$ pixels, and it comprises of $16$ mutually exclusive vegetation classes. The spatial resolution is 20 meters per pixel (MPP). The detailed class description and ground truth maps are presented in Figure <ref>. Moreover, the disjoint Training/Test sample maps are presented in Figures <ref> and <ref>.

GT Maps

Disjoint Training

Disjoint Test

=1mm =1mm
Background     10776      =1mm =1mm      Alfalfa     46     =1mm =1mm      Corn notill     1428
=1mm =1mm

Corn min     830     =1mm =1mm      Corn     237     =1mm =1mm      Grass/Pasture     483
=1mm =1mm

Grass/Trees     730     =1mm =1mm      Grass/pasture-mowed     28     =1mm =1mm      Hay windrowed     478
=1mm =1mm

Oats     20     =1mm =1mm      Soybeans notill     972     =1mm =1mm      Soybeans min     2455
=1mm =1mm

Soybean clean     593     =1mm =1mm      Wheat     205     =1mm =1mm      Woods     1265
=1mm =1mm

Bldg Grass Tree Drives     386     =1mm =1mm      Stone steel towers     93         =1mm =1mm Total samples     21025

The type associated with the land-cover classes and number of available samples in the Indian Pines (IP) dataset. Moreover, Spatially disjoint training and test samples for the IP dataset are also presented.

The Kennedy Space Center (KSC) dataset was gathered in 1996 by AVIRIS [317], with wavelengths ranging from $400$ to $2500$ $nm$. The image has $512\times{614}$ pixels and $176$ spectral bands after removal of some low signal-to-noise ratio (SNR) bands. The KSC dataset comprises $5202$ labeled samples, with a total of $13$ upland and wetland classes. The detailed class description and ground truth maps are presented in Figure <ref>. Moreover, the disjoint Training/Test sample maps are presented in Figures <ref> and <ref>.

GT Maps

Disjoint Training

Disjoint Test

=1mm =1mm      Background     309157     =1mm =1mm       Scrub     761      =1mm =1mm      Willow swamp     243
=1mm =1mm

CP hammock     256     =1mm =1mm      Slash pine     252     =1mm =1mm      Oak/Broadleaf     161
=1mm =1mm

Hardwood     229     =1mm =1mm      Swap     105     =1mm =1mm      Graminoid marsh     431
=1mm =1mm

Spartina marsh     520     =1mm =1mm      Cattail marsh     404     =1mm =1mm      Salt marsh     419
=1mm =1mm

Mud flats     503     =1mm =1mm      Water     927         =1mm =1mm Total samples     207400

The type associated with the land-cover classes and number of available samples in the Kennedy Space Center (KSC) dataset. Moreover, Spatially disjoint training and test samples for the KSC dataset are also presented.

The University of Pavia (UP) dataset was acquired by the Reflective Optics System Imaging Spectrometer (ROSIS) sensor during a flight campaign over the university campus at Pavia, Northern Italy [318]. It consists of $610\times{340}$ pixels with $103$ spectral bands in the wavelength range from $430$ to $860~{nm}$ and 2.5 MPP. It comprises 9 urban land-cover classes. The detailed class description and ground truth maps are presented in Figure <ref>. Moreover, the disjoint Training/Test sample maps are presented in Figures <ref> and <ref>.

GT Maps

Disjoint Training

Disjoint Test

=1mm =1mm
Background     164624      =1mm =1mm      Asphalt     6631     =1mm =1mm      Meadows     18649
=1mm =1mm

Gravel     2099     =1mm =1mm      Trees     3064     =1mm =1mm      Painted metal sheets     1345
=1mm =1mm

Bare Soil     5029     =1mm =1mm      Bitumen     1330     =1mm =1mm      Self Blocking Bricks     3682
=1mm =1mm
Shadows     947         =1mm =1mm Total samples     207400 =1mm =1mm

The type associated with the land-cover classes and number of available samples in the Pavia University (PU) dataset. Moreover, Spatially disjoint training and test samples for the PU dataset are also presented.

The IEEE Geoscience and Remote Sensing Society published the University of Houston (UH) dataset–collected by the Compact Airborne Spectrographic Imager (CASI)– in 2013 [319], as part of its Data Fusion Contest. It is composed of $340\times{1905}$ pixels with 144 spectral bands. The spatial resolution of this dataset is $2.5$ MPP with a wavelength ranging from $0.38$ to $1.05$ $\mu$m. Finally, the ground truth comprises 15 different land-cover classes. The detailed class description and ground truth maps are presented in Figure <ref> and disjoint Training/Test sample maps are presented in Figures <ref> and <ref>.

GT Maps

Disjoint Training

Disjoint Testing

=1mm =1mm      Background     1314661      =1mm =1mm      Grass-healthy     1251     =1mm =1mm      Grass-stressed     1254
=1mm =1mm

Grass-synthetic     697      =1mm =1mm      Tree     1244     =1mm =1mm      Soil     1242
=1mm =1mm

Water     325       =1mm =1mm      Residential     1286     =1mm =1mm      Commercial     1244
=1mm =1mm

Road     1252      =1mm =1mm      Highway     1227     =1mm =1mm      Railway     1235
=1mm =1mm

Parking-lot1     1233      =1mm =1mm      Parking-lot2     469     =1mm =1mm      Tennis-court     428
=1mm =1mm

Running-track     660          Total samples     1329690

The type associated with the land-cover classes and number of available samples in the Houston (UH) dataset.

The University of Trento (UT) dataset was gathered by the using AISA eagle sensor over the rural regions in the south of Trento, Italy. The HSI contains 63 spectral bands within a wavelength of range $0.42$ to $0.99$ $\mu{m}$ [320]. The scene has $600 \times 166$ pixels, which comprises of $6$ mutually exclusive vegetation land-cover classes where the spectral resolution is 9.2 $nm$, and the spatial resolution is 1 meter per pixel (MPP). In addition, the available samples are divided into disjoint training and test samples of 6 classes and Fig. <ref> lists the information about the per class number of samples for six different land-covers.

GT Maps

Disjoint Training

Disjoint Testing

0.90! =1mm =1mm
Background     168986     =1mm =1mm
Apples     4034

=1mm =1mm

Buildings     2903     =1mm =1mm
Ground     479

=1mm =1mm

Woods     9123     =1mm =1mm
Vineyard     10501

=1mm =1mm

Roads     3174
=1mm =1mm Total samples     199200

The type associated with the land-cover classes and number of available samples in the University of Trento (UT) dataset.

Table <ref> provides a summary description of each dataset used in the following experiments whereas, Table <ref> enlists the numbers of disjoint samples (i.e., Train/Test samples selected from each class) used for all the experimental results. Please note that the number of train/test (i.e. percentage) samples and geographical locations of train/test samples remain the same for all experimental methods (competing methods).

Summary of the HSI datasets used for experimental evaluation.
! —     IP     PU     KSC     UH     UT
Year     1992     2001     1996     2013

Source     AVIRIS     ROSIS-03     AVIRIS     CASI     AISA

Spatial     $145\times 145$     $610 \times 610$     $512\times 614$     $340\times 1905$     $600 \times 166$

Spectral     220     115     176     144     63

Wavelength     $400-2500$     $430-860$     $400-2500$     $0.35-1.05$     $0.42-0.99$

Samples     21025     207400     314368     1329690     199200

Classes     16     9     13     15     6

Sensor     Aerial     Aerial     Aerial     Aerial     Aerial

Resolution     $20~m$     $1.3~m$     $10~nm$     $2.5~mpp$     $1~mpp$

Number of Disjoint Train/Test Samples used for the experimental results. Where TrS and TeS stands for disjoint Train and Test samples, respectively.
4c|IP Data     4c|KSC Data     4c|PU Data     4c|UH Data
4cUT Data
Class     Land Cover     TrS     TeS     Class     Land Cover     TrS     TeS     Class     Land Cover     TrS     TeS     Class     Land Cover     TrS     TeS     Class     Land Cover     TrS     TeS
1          29     25      1           114       602      1          548     6083     1          198     1053          Apples     129     3905

2          762     675        2           36     207     2          540      18109     2           190     1064           Buildings     125      2778

3          435     404        3           38      218     3          392     1707     3          192     505           Ground     105       374

4          146     99       4           38      214     4          524     2540     4          188     1056           Woods     154       8969

5          232     274       5           24     137     5          265     1080     5          186     1056           Vineyard     184       10317

6          394     354       6           34      195     6          532     4497     6          182     143          Roads     122       3052

7          16     2      7           16      89      7          375     955     7          196     1072

8          235     250       8           65     366      8          514     3168     8          191     1053

9          10     10       9           78     442      9          231     716     9          193     1059

10          470     503       10           61     343                         10          191     1036

11          1424     1065       11           63     356                         11          181     1054

12          328     282       12           75     428                         12          192     1041

13          132     80       13           139     788                          13          184     285

14          728     545                                     14          181     247

15          291     99                                     15          187     473

16           57     44

Classification results obtained by RF [75], MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326], HybridSN [213], and MorphCNN [196] on the disjoint train-test dataset for the PU scene.
Class     RF [75]     MLR [321]     SVM [322]    MLP [69]    RNN [70]    LSTM [323]    GRU [324]    CNN-1D [218]    CNN-2D [325]    CNN-3D [326]    HybridSN [213]     MorphCNN [196]

1     89.98$\pm$0.15     77.68$\pm$0.0     82.23$\pm$0.0     84.53$\pm$1.89     83.08$\pm$3.3     82.63$\pm$2.39     77.25$\pm$6.92     87.18$\pm$2.11     93.4$\pm$1.89     85.66$\pm$4.0     89.74$\pm$5.19     94.52$\pm$1.9

2     74.39$\pm$0.01     58.79$\pm$0.01     65.81$\pm$0.0     75.13$\pm$2.4     67.9$\pm$2.92     78.74$\pm$1.99     80.1$\pm$5.12     89.64$\pm$2.53     96.84$\pm$1.93     95.88$\pm$1.71     81.78$\pm$3.15     97.12$\pm$8.71

3     38.42$\pm$0.13     67.21$\pm$0.02     66.72$\pm$0.0     68.37$\pm$5.17     65.17$\pm$7.99     60.73$\pm$11.0     54.79$\pm$14.82     71.1$\pm$5.98     65.48$\pm$13.94     68.11$\pm$6.47     82.88$\pm$1.54     85.08$\pm$4.53

4     98.24$\pm$0.05     74.27$\pm$0.05     97.77$\pm$0.0     93.5$\pm$2.32     90.72$\pm$2.56     97.1$\pm$1.22     92.05$\pm$2.31     95.32$\pm$1.49     95.55$\pm$2.14     97.02$\pm$0.83     83.66$\pm$4.58     97.0$\pm$1.03

5     95.98$\pm$0.04     98.88$\pm$0.04     99.37$\pm$0.0     99.37$\pm$0.08     99.23$\pm$0.09     99.28$\pm$0.08     99.51$\pm$0.12     99.48$\pm$0.26     98.03$\pm$0.92     98.9$\pm$0.56     99.94$\pm$0.04     99.25$\pm$0.22

6     51.43$\pm$0.19     93.53$\pm$0.02     91.62$\pm$0.0     89.94$\pm$4.14     85.07$\pm$3.14     65.94$\pm$5.92     74.86$\pm$11.38     88.28$\pm$2.33     80.52$\pm$9.39     68.85$\pm$11.29     72.43$\pm$13.23     93.92$\pm$3.88

7     80.63$\pm$0.36     85.08$\pm$0.05     87.36$\pm$0.0     87.2$\pm$3.05     82.94$\pm$3.79     84.95$\pm$4.02     90.17$\pm$3.9     86.77$\pm$3.38     89.29$\pm$9.48     73.09$\pm$9.53     96.16$\pm$1.88     84.98$\pm$10.74

8     97.64$\pm$0.14     87.58$\pm$0.01     90.46$\pm$0.0     90.37$\pm$1.24     85.85$\pm$4.97     88.89$\pm$7.83     90.42$\pm$4.39     90.43$\pm$3.34     94.5$\pm$5.44     95.21$\pm$1.69     92.80$\pm$0.90     96.62$\pm$2.21

9     94.92$\pm$0.05     99.22$\pm$0.05     93.71$\pm$0.0     98.44$\pm$1.17     94.52$\pm$4.79     98.29$\pm$1.47     93.51$\pm$7.93     97.33$\pm$3.31     95.8$\pm$0.76     93.54$\pm$1.76     94.04$\pm$3.99     97.05$\pm$0.46

OA     77.44$\pm$0.06     72.23$\pm$0.0     77.8$\pm$0.0     82.05$\pm$0.88     77.07$\pm$0.95     80.38$\pm$0.52     80.7$\pm$0.56     89.09$\pm$0.97     92.55$\pm$1.02     89.43$\pm$1.37     84.18$\pm$1.40     95.51$\pm$0.66

AA     80.18$\pm$0.06     82.47$\pm$0.01     86.12$\pm$0.0     87.43$\pm$1.03     83.83$\pm$0.72     84.06$\pm$0.74     83.63$\pm$2.03     89.5$\pm$1.03     89.94$\pm$1.37     86.25$\pm$1.98     88.16$\pm$1.94     93.95$\pm$0.96

k(x100)     70.44$\pm$0.07     65.44$\pm$0.0     72.06$\pm$0.0     76.89$\pm$1.07     70.84$\pm$1.04     74.32$\pm$0.68     74.76$\pm$1.02     85.5$\pm$1.22     89.9$\pm$1.42     85.61$\pm$1.94     79.13$\pm$1.42     93.95$\pm$0.88

a) 1PC
b) GT
c) MLR
d) SVM
e) MLP
f) RNN

g) LSTM
h) GRU
i) CNN1D
j) CNN2D
k) CNN3D
l) MorphCNN (90.55%)

Classification Maps obtained by MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326] and MorphCNN [196] on the disjoint train-test dataset for the UP scene.

§.§ Experimental Results on Disjoint Train/Test Samples

To strengthen the ideas highlighted in this survey and to make the claims valid, the main contributions made in recent years include MLR, SVM, MLP, RNN, LSTM, GRU, CNN-1D, CNN-2D, CNN-3D, and MorphCNN have been considered to compare the experimental results. Some of the representative works for each above are as follows; Cloud Implementation of Logistic Regression for HSIC [321, 327, 328] (MLR), Classification of Hyperspectral Remote Sensing Images with SVM [322], (SVM), Deep Recurrent Neural Networks for HSIC [70] (RNN), Long Short-Term Memory [323] (LSTM), On the properties of Neural Machine Translation: Encoder-Decoder Approaches [324] (GRU), Deep Convolutional Neural Networks for HSIC [218] (CNN1D), Deep Supervised Learning for Hyperspectral Data Classification through Convolutional Neural Networks [325] (CNN2D), 3-D Deep Learning Approach for Remote Sensing Image Classification [326] (CNN3D), Morphological Convolutional Neural Networks for HSIC [196] (MorphCNN), and MLP [69].

To some extent, all the aforesaid works are based on Convolutional and Recurrent Networks and are evaluated on four benchmark HSI datasets namely IP, PU, KSC, Houston Scene, and the University of Toronto. This survey only pays attention to the robustness of all these models while considering the small sample size of training data to classify HSI for joint spatial-spectral classification.

Classification results obtained by RF [75], MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326], HybridSN [213], and MorphCNN [196] on the disjoint train-test dataset for the IP scene.
Class     RF [75]     MLR [321]     SVM [322]    MLP [69]    RNN [70]    LSTM [323]    GRU [324]    CNN-1D [218]    CNN-2D [325]    CNN-3D [326]    HybridSN [213]     MorphCNN [196]

1     85.33$\pm$1.88     80.0$\pm$0.0     88.0$\pm$0.0     73.6$\pm$7.42     58.4$\pm$4.8     89.6$\pm$1.96     77.6$\pm$9.33     80.8$\pm$12.75     73.64$\pm$14.77     48.18$\pm$22.84     82.66$\pm$13.59     92.27$\pm$3.55

2     55.11$\pm$0.32     81.48$\pm$0.0     80.0$\pm$0.0     81.45$\pm$1.07     75.5$\pm$1.48     82.22$\pm$1.26     81.1$\pm$2.77     79.38$\pm$4.16     83.12$\pm$6.12     85.12$\pm$7.88     82.17$\pm$2.64     84.05$\pm$8.03

3     22.77$\pm$0.20     54.11$\pm$0.12     69.55$\pm$0.0     64.55$\pm$2.85     63.37$\pm$1.93     64.16$\pm$5.44     70.35$\pm$1.36     74.26$\pm$6.12     81.98$\pm$3.89     77.22$\pm$13.04     76.73$\pm$4.02     79.34$\pm$3.45

4     13.13$\pm$1.64     38.38$\pm$0.0     48.48$\pm$0.0     47.07$\pm$10.41     29.49$\pm$5.4     55.35$\pm$10.62     53.33$\pm$11.15     31.92$\pm$11.55     45.39$\pm$6.36     50.11$\pm$10.04     33.33$\pm$3.59     52.14$\pm$6.24

5     41.60$\pm$0.78     91.97$\pm$0.0     87.23$\pm$0.0     86.94$\pm$1.07     87.59$\pm$1.5     89.27$\pm$0.68     88.4$\pm$0.85     90.73$\pm$1.07     89.11$\pm$5.55     80.28$\pm$6.52     81.14$\pm$8.89     91.66$\pm$1.69

6     94.06$\pm$0.23     94.63$\pm$0.0     96.33$\pm$0.0     95.93$\pm$0.97     95.31$\pm$0.83     96.39$\pm$0.87     96.38$\pm$1.14     96.39$\pm$0.9     95.02$\pm$5.68     89.81$\pm$4.03     97.36$\pm$2.54     95.74$\pm$2.28

7     0.0$\pm$0.0     0.0$\pm$0.0     50.0$\pm$0.0     10.0$\pm$20.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0     0.0$\pm$0.0

8     91.33$\pm$0.18     100.0$\pm$0.0     100.0$\pm$0.0     99.84$\pm$0.2     99.52$\pm$0.3     99.2$\pm$0.91     99.12$\pm$0.64     99.84$\pm$0.32     99.96$\pm$0.13     95.96$\pm$6.78     96.53$\pm$3.78     100.0$\pm$0.0

9     40.0$\pm$0.0     0.0$\pm$0.0     50.0$\pm$0.0     80.0$\pm$15.49     56.0$\pm$10.2     76.0$\pm$8.0     66.0$\pm$4.9     50.0$\pm$8.94     26.66$\pm$15.87     77.78$\pm$21.66     66.66$\pm$24.94     44.44$\pm$19.88

10     26.83$\pm$1.26     66.76$\pm$0.08     76.54$\pm$0.0     75.35$\pm$5.02     71.13$\pm$5.93     81.51$\pm$2.76     78.53$\pm$4.64     81.83$\pm$4.69     77.44$\pm$8.99     77.9$\pm$6.2     74.35$\pm$9.46     80.77$\pm$3.77

11     81.06$\pm$0.49     84.13$\pm$0.0     87.7$\pm$0.0     83.19$\pm$1.51     78.86$\pm$1.45     80.4$\pm$2.43     82.29$\pm$1.82     80.39$\pm$3.65     89.4$\pm$5.47     82.73$\pm$3.81     79.18$\pm$4.92     88.54$\pm$5.03

12     28.95$\pm$0.44     66.31$\pm$0.0     77.3$\pm$0.0     78.58$\pm$2.95     71.91$\pm$5.07     76.31$\pm$1.39     83.19$\pm$1.16     84.75$\pm$7.5     87.72$\pm$3.06     82.64$\pm$14.49     71.04$\pm$4.11     88.46$\pm$4.26

13     86.25$\pm$1.02     95.0$\pm$0.0     97.5$\pm$0.0     98.0$\pm$0.61     97.0$\pm$1.7     97.25$\pm$0.94     97.75$\pm$0.94     97.75$\pm$0.5     95.28$\pm$4.74     89.72$\pm$6.89     96.25$\pm$4.44     87.64$\pm$3.43

14     91.07$\pm$0.87     90.64$\pm$0.0     91.38$\pm$0.0     92.92$\pm$1.48     90.28$\pm$1.09     94.13$\pm$1.18     92.88$\pm$1.79     93.32$\pm$2.34     98.94$\pm$0.55     98.31$\pm$1.41     91.68$\pm$4.33     98.82$\pm$1.01

15     10.10$\pm$0.0     89.9$\pm$0.0     80.81$\pm$0.0     87.88$\pm$3.78     75.56$\pm$6.43     90.71$\pm$2.34     93.54$\pm$1.64     89.9$\pm$4.78     82.02$\pm$14.83     55.17$\pm$27.57     45.45$\pm$21.39     69.44$\pm$15.86

16     71.96$\pm$3.86     97.73$\pm$0.0     97.73$\pm$0.0     87.27$\pm$4.45     88.64$\pm$4.31     94.09$\pm$2.32     95.45$\pm$2.49     96.82$\pm$2.32     82.0$\pm$6.69     82.5$\pm$12.5     84.09$\pm$4.90     84.0$\pm$4.21

OA     60.80$\pm$0.14     80.33$\pm$0.02     84.12$\pm$0.0     82.95$\pm$0.23     79.07$\pm$0.33     83.55$\pm$0.39     84.2$\pm$0.21     84.0$\pm$0.28     87.25$\pm$1.03     83.6$\pm$1.41     80.86$\pm$1.74     87.45$\pm$1.01

AA     52.47$\pm$0.21     70.69$\pm$0.01     79.91$\pm$0.0     77.66$\pm$1.98     71.16$\pm$0.87     79.16$\pm$0.75     78.49$\pm$0.36     76.76$\pm$0.75     75.48$\pm$2.12     73.34$\pm$3.46     72.41$\pm$2.32     77.33$\pm$1.56

k(x100)     54.41$\pm$0.18     77.47$\pm$0.02     81.87$\pm$0.0     80.56$\pm$0.26     76.12$\pm$0.4     81.27$\pm$0.44     82.01$\pm$0.26     81.81$\pm$0.35     85.48$\pm$1.15     81.36$\pm$1.62     78.24$\pm$1.98     85.75$\pm$1.14

a) 1PC
b) GT
c) MLR
d) SVM
e) MLP
f) RNN

g) LSTM
h) GRU
i) CNN1D
j) CNN2D
k) CNN3D
l) MorphCNN

Classification Maps obtained by MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326] and MorphCNN [196] on the disjoint train-test dataset for the IP scene.

Classification results obtained by RF [75], MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326], HybridSN [213], and MorphCNN [196] on the disjoint train-test dataset for the UH scene.

Class     RF [75]     MLR [321]     SVM [322]    MLP [69]    RNN [70]    LSTM [323]    GRU [324]    CNN-1D [218]    CNN-2D [325]    CNN-3D [326]    HybridSN [213]     MorphCNN [196]

1     82.87$\pm$0.04     82.24$\pm$0.06     82.34$\pm$0.0     81.23$\pm$0.28     82.22$\pm$0.28     82.76$\pm$0.36     82.58$\pm$0.35     82.28$\pm$0.98     82.25$\pm$0.65     82.1$\pm$0.39     82.74$\pm$0.29     82.43$\pm$0.33

2     82.51$\pm$0.38     82.5$\pm$0.07     83.36$\pm$0.0     82.29$\pm$0.55     82.87$\pm$0.33     80.19$\pm$1.36     81.64$\pm$0.71     91.78$\pm$6.46     84.15$\pm$0.28     84.14$\pm$0.45     90.91$\pm$6.22     84.42$\pm$0.19

3     64.09$\pm$0.49     99.8$\pm$0.0     99.8$\pm$0.0     99.72$\pm$0.1     99.72$\pm$0.2     99.68$\pm$0.16     99.88$\pm$0.1     99.92$\pm$0.16     90.31$\pm$4.41     77.85$\pm$4.8     98.81$\pm$0.74     97.21$\pm$1.23

4     92.04$\pm$0.0     98.3$\pm$0.0     98.96$\pm$0.0     87.58$\pm$1.28     93.5$\pm$2.02     91.23$\pm$1.29     93.22$\pm$2.84     94.36$\pm$3.12     87.24$\pm$3.21     89.24$\pm$1.1     83.96$\pm$1.24     92.37$\pm$0.33

5     99.81$\pm$0.07     97.44$\pm$0.0     98.77$\pm$0.0     97.35$\pm$0.49     97.76$\pm$0.29     97.65$\pm$0.31     97.37$\pm$0.16     98.77$\pm$0.13     99.51$\pm$0.48     98.97$\pm$0.59     99.46$\pm$0.75     99.77$\pm$0.53

6     96.27$\pm$0.32     94.41$\pm$0.0     97.9$\pm$0.0     94.55$\pm$0.28     95.1$\pm$0.0     97.06$\pm$1.79     98.32$\pm$1.63     95.8$\pm$1.88     96.43$\pm$2.14     98.91$\pm$1.44     98.60$\pm$1.97     99.46$\pm$1.15

7     86.19$\pm$0.34     73.37$\pm$0.07     77.43$\pm$0.0     75.24$\pm$2.27     81.4$\pm$0.43     78.88$\pm$1.0     77.03$\pm$2.18     82.78$\pm$2.23     86.44$\pm$2.18     85.48$\pm$1.98     75.62$\pm$3.89     88.07$\pm$1.78

8     41.69$\pm$0.23     63.82$\pm$0.0     60.3$\pm$0.0     57.0$\pm$6.97     40.06$\pm$1.07     40.11$\pm$1.92     53.62$\pm$2.97     75.5$\pm$6.71     70.03$\pm$3.96     62.06$\pm$3.01     93.16$\pm$0.20     73.09$\pm$3.5

9     86.02$\pm$0.48     70.23$\pm$0.04     76.77$\pm$0.0     75.58$\pm$2.86     76.54$\pm$2.96     81.55$\pm$4.12     79.06$\pm$1.61     81.44$\pm$2.0     79.53$\pm$6.38     80.81$\pm$4.32     81.39$\pm$5.24     84.09$\pm$2.73

10     36.00$\pm$0.0     55.6$\pm$0.0     61.29$\pm$0.0     48.78$\pm$2.27     47.44$\pm$1.44     47.37$\pm$2.29     49.54$\pm$2.61     68.71$\pm$14.55     60.22$\pm$4.2     54.75$\pm$4.63     76.51$\pm$10.40     62.86$\pm$3.08

11     64.67$\pm$0.16     74.21$\pm$0.04     80.55$\pm$0.0     76.25$\pm$0.46     76.24$\pm$0.81     76.38$\pm$1.09     80.82$\pm$0.71     85.24$\pm$2.83     82.93$\pm$7.68     66.78$\pm$3.34     89.21$\pm$5.62     89.15$\pm$6.86

12     67.27$\pm$0.09     70.41$\pm$0.0     79.92$\pm$0.0     75.31$\pm$3.75     76.33$\pm$3.09     79.98$\pm$3.32     84.15$\pm$3.13     89.93$\pm$4.29     92.87$\pm$3.31     93.83$\pm$1.92     96.28$\pm$2.29     93.02$\pm$3.32

13     89.23$\pm$0.43     67.72$\pm$0.0     70.88$\pm$0.0     73.19$\pm$2.15     69.12$\pm$1.61     71.37$\pm$3.54     72.63$\pm$3.68     74.88$\pm$5.14     86.21$\pm$2.65     82.34$\pm$2.49     86.78$\pm$6.67     89.61$\pm$1.34

14     100.0$\pm$0.0     98.79$\pm$0.0     100.0$\pm$0.0     99.84$\pm$0.32     100.0$\pm$0.0     99.11$\pm$0.47     99.92$\pm$0.16     99.68$\pm$0.16     98.92$\pm$1.8     96.31$\pm$3.67     100.0$\pm$0.0     99.19$\pm$1.3

15     90.06$\pm$0.45     95.56$\pm$0.0     96.41$\pm$0.0     97.8$\pm$0.51     97.59$\pm$0.47     98.14$\pm$0.31     98.22$\pm$0.59     98.48$\pm$0.24     77.63$\pm$2.91     75.85$\pm$2.69     100.0$\pm$0.0     97.04$\pm$4.47

OA     75.38$\pm$0.06     78.97$\pm$0.01     81.86$\pm$0.0     78.22$\pm$0.36     77.95$\pm$0.68     78.16$\pm$0.28     80.21$\pm$0.27     86.42$\pm$1.64     83.27$\pm$0.8     80.24$\pm$0.55     88.31$\pm$1.78     86.51$\pm$0.71

AA     78.58$\pm$0.11     81.63$\pm$0.01     84.31$\pm$0.0     81.45$\pm$0.37     81.06$\pm$0.55     81.43$\pm$0.32     83.2$\pm$0.27     87.97$\pm$1.38     84.98$\pm$0.74     81.96$\pm$0.75     90.23$\pm$1.39     88.78$\pm$0.68

k(x100)     73.49$\pm$0.07     77.3$\pm$0.01     80.43$\pm$0.0     76.55$\pm$0.39     76.23$\pm$0.71     76.52$\pm$0.3     78.66$\pm$0.29     85.27$\pm$1.77     81.89$\pm$0.86     78.62$\pm$0.59     87.33$\pm$1.92     85.4$\pm$0.76

a) 1PC
b) GT
c) MLR

d) SVM
e) MLP
f) RNN

g) LSTM
h) GRU
i) CNN1D

j) CNN2D
k) CNN3D
l) MorphCNN

Classification Maps obtained by MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326] and MorphCNN [196] on the disjoint train-test dataset for the UH scene.

Classification results obtained by RF [75], MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326], HybridSN [213], and MorphCNN [196] on the disjoint train-test dataset for the KSC scene.
Class     RF [75]     MLR [321]     SVM [322]    MLP [69]    RNN [70]    LSTM [323]    GRU [324]    CNN-1D [218]    CNN-2D [325]    CNN-3D [326]    HybridSN [213]     MorphCNN [196]

1     99.69$\pm$0.0     100.0$\pm$0.00     94.13$\pm$0.00     99.18$\pm$1.17     87.33$\pm$1.73     92.22$\pm$0.91     89.44$\pm$0.69     99.79$\pm$0.14     85.52$\pm$15.45     97.17$\pm$2.93     100.0$\pm$0.0     97.63$\pm$2.26

2     98.38$\pm$0.22     99.03$\pm$0.001     0.00$\pm$0.00     86.63$\pm$7.33     63.12$\pm$1.49     81.64$\pm$4.17     70.85$\pm$0.91     99.19$\pm$1.14     67.31$\pm$25.41     92.91$\pm$2.62     100.0$\pm$0.0     86.79$\pm$9.64

3     99.23$\pm$0.57     99.54$\pm$0.00     54.59$\pm$0.00      84.25$\pm$2.70     69.72$\pm$2.82     75.38$\pm$0.21     78.89$\pm$7.79     95.11$\pm$3.05     60.09$\pm$16.23     81.04$\pm$12.05     99.69$\pm$0.21     98.31$\pm$2.37

4     88.16$\pm$1.22     99.06$\pm$0.002     17.28$\pm$0.00      78.97$\pm$11.86     47.82$\pm$4.20     58.09$\pm$1.16     44.08$\pm$3.41     77.73$\pm$2.81     45.17$\pm$9.84     44.54$\pm$16.19     99.53$\pm$0.66     88.94$\pm$15.64

5     73.72$\pm$0.0     100.0$\pm$0.00     0.00$\pm$0.00     13.38$\pm$18.92     68.37$\pm$5.63     74.21$\pm$5.20     65.21$\pm$3.28     80.53$\pm$4.85     67.40$\pm$12.84     85.15$\pm$6.98     98.78$\pm$0.34     48.66$\pm$34.29

6     88.88$\pm$0.24     100.0$\pm$0.00     0.00$\pm$0.00      78.12$\pm$8.79     56.24$\pm$1.97     65.12$\pm$6.03     59.82$\pm$2.30     91.97$\pm$1.74     65.47$\pm$29.63     62.74$\pm$15.45     100.0$\pm$0.0     86.32$\pm$16.43

7     100.0$\pm$0.0     89.88$\pm$0.00     0.00$\pm$0.00      78.65$\pm$3.99     83.52$\pm$8.91     90.26$\pm$1.40     89.14$\pm$3.47     95.13$\pm$1.40     77.15$\pm$28.34     80.52$\pm$17.07     97.75$\pm$1.83     97.75$\pm$3.17

8     85.51$\pm$0.22     100.0$\pm$0.00     60.10$\pm$0.00      89.62$\pm$8.25     65.57$\pm$2.32     71.40$\pm$2.99     69.76$\pm$2.53     97.45$\pm$0.51     64.75$\pm$13.87     71.49$\pm$11.96     99.90$\pm$0.12     70.76$\pm$34.44

9     96.68$\pm$0.42     100.0$\pm$0.00     89.37$\pm$0.00     97.59$\pm$1.79     88.39$\pm$3.35     90.72$\pm$2.93     86.72$\pm$1.36     99.92$\pm$0.11     89.22$\pm$8.06     98.94$\pm$1.33     100.0$\pm$0.0     91.93$\pm$7.16

10     99.22$\pm$0.13     100.0$\pm$0.00     98.83$\pm$0.00     96.50$\pm$3.33     92.42$\pm$3.12     88.92$\pm$3.37     88.53$\pm$1.58     99.90$\pm$0.13     73.08$\pm$23.37     90.67$\pm$8.13     100.0$\pm$0.0     100.0$\pm$0.00

11     100.0$\pm$0.0     98.03$\pm$0.001     94.94$\pm$0.00     98.50$\pm$0.86     83.89$\pm$2.29     90.26$\pm$1.08     84.83$\pm$3.64     100.0$\pm$0.0     87.55$\pm$10.06     97.56$\pm$1.17     96.34$\pm$0.79     100.0$\pm$0.0

12     97.89$\pm$0.19     99.29$\pm$0.001     89.25$\pm$0.00     98.52$\pm$0.79     81.31$\pm$4.48     87.46$\pm$2.05     83.57$\pm$4.87     98.36$\pm$1.06     82.48$\pm$19.17     99.30$\pm$0.99     99.06$\pm$0.99     97.89$\pm$2.06

13     100.0$\pm$0.0     100.0$\pm$0.0     100.0$\pm$0.0     100.0$\pm$0.0     99.88$\pm$0.10     100.0$\pm$0.0     99.92$\pm$0.05     100.0$\pm$0.0     99.92$\pm$0.12     100.0$\pm$0.0     100.0$\pm$0.0     100.0$\pm$0.0

OA     96.17$\pm$0.07     99.45$\pm$0.001     72.84$\pm$0.00      91.76$\pm$0.56     81.47$\pm$1.17     86.10$\pm$0.40     82.76$\pm$0.72     97.18$\pm$0.18     79.98$\pm$13.38     89.71$\pm$1.30     99.48$\pm$0.05     92.76$\pm$2.08

AA     94.41$\pm$0.08     98.83$\pm$0.001     53.73$\pm$0.00     84.61$\pm$0.62     75.96$\pm$1.71     81.97$\pm$0.34     77.75$\pm$0.70     95.00$\pm$0.34     74.24$\pm$16.27     84.77$\pm$2.60     99.31$\pm$0.17     89.61$\pm$1.60

k(x100)     95.74$\pm$0.08     99.40$\pm$0.001     69.29$\pm$0.00     90.82$\pm$0.62     79.33$\pm$1.30     84.51$\pm$0.45     80.79$\pm$0.80     96.86$\pm$0.20     77.63$\pm$14.99     88.51$\pm$1.46     99.43$\pm$0.06     91.94$\pm$2.31

a) 1PC
b) GT
c) MLR
d) SVM
e) MLP
f) RNN

g) LSTM
h) GRU
i) CNN1D
j) CNN2D
k) CNN3D
l) MorphCNN

Classification Maps obtained by MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326] and MorphCNN [196] on the disjoint train-test dataset for the KSC scene.

Here we have enlisted the experimental results with detailed discussion on the obtained results. The obtained accuracies for disjoint training and test samples are shown in Tables <ref>, <ref>, <ref> and <ref> and Figures <ref>, <ref> <ref>, and <ref>. All the results shown in the Tables and Figures are obtained using the 10-cross-validation process to compute the overall, average and kappa $(\kappa)$ accuracy for comparison purposes. For instance, let us assume the case of Pavia University results, for this particular case, the work [196] has the highest average, overall and kappa $(\kappa)$ accuracies which are 95.51%, 93.95%, and 93.95% respectively in comparison with the average, overall and kappa $(\kappa)$ accuracies for other comparative works; 92.55%, 89.94%, 89.9% for [325], 89.43%, 86.25%, 85.61% for [326], 89.09%, 89.5%, 85.5% for [218], 82.05%, 87.43%, 76.89% for [69], 80.38%, 83.63%, 74.76% for [324], 80.38%, 84.06%, 74.32% for [323], 77.8%, 86.12%, 72.06% for [322], 77.07%, 83.83%, 70.84% for [70], and 72.23%, 82.12%, 65.44% for [321]. Similar observations can be made of the other experimental datasets.

Classification results obtained by RF [75], MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326], HybridSN [213], and MorphCNN [196] on the disjoint train-test dataset for the University of Trento (UT) scene.
Class     RF [75]     MLR [321]     SVM [322]    MLP [69]    RNN [70]    LSTM [323]    GRU [324]    CNN-1D [218]    CNN-2D [325]    CNN-3D [326]    HybridSN [213]     MorphCNN [196]

1     97.27$\pm$0.33     92.57$\pm$0.00     95.03$\pm$0.00     70.68$\pm$3.69     93.46$\pm$3.67     95.71$\pm$0.32     95.78$\pm$0.97     98.06$\pm$0.72     96.35$\pm$0.22     92.64$\pm$7.45     99.15$\pm$0.16     97.83$\pm$0.37

2     89.50$\pm$0.11     90.92$\pm$0.00     88.66$\pm$0.00     76.22$\pm$2.90     83.09$\pm$1.50     84.73$\pm$0.87     84.40$\pm$1.51     89.16$\pm$1.99     85.87$\pm$1.87     75.70$\pm$7.23     81.65$\pm$2.93     90.41$\pm$1.83

3     75.04$\pm$0.33     90.10$\pm$0.00     91.71$\pm$0.00     83.95$\pm$2.08     69.51$\pm$0.87     59.26$\pm$14.31     65.15$\pm$4.84     70.67$\pm$8.99     66.84$\pm$2.00     60.07$\pm$3.57     74.86$\pm$10.58     83.86$\pm$10.69

4     99.96$\pm$0.01     92.77$\pm$0.00     85.18$\pm$0.00     37.55$\pm$2.74     98.44$\pm$0.97     97.72$\pm$1.41     99.65$\pm$0.18     99.86$\pm$0.11     98.67$\pm$0.61     98.88$\pm$0.80     99.64$\pm$0.28     99.06$\pm$0.57

5     99.97$\pm$0.00     99.14$\pm$0.00     97.76$\pm$0.00     99.98$\pm$.01     98.12$\pm$0.56     96.34$\pm$0.80     98.20$\pm$0.54     99.78$\pm$0.19     99.32$\pm$0.39     98.40$\pm$1.77     98.12$\pm$1.79     99.87$\pm$0.14

6     67.71$\pm$0.31     66.80$\pm$0.00     72.37$\pm$0.00     73.91$\pm$2.88     65.54$\pm$0.90     69.04$\pm$2.29     63.89$\pm$1.02     77.94$\pm$3.83     74.34$\pm$6.48     69.42$\pm$3.43     70.72$\pm$4.74     82.64$\pm$0.88

OA     94.95$\pm$0.07     92.08$\pm$0.00     89.98$\pm$0.00     63.51$\pm$9.43     92.43$\pm$0.34     92.27$\pm$0.72     93.03$\pm$0.28     95.94$\pm$0.23     94.45$\pm$0.13     92.14$\pm$1.05     94.03$\pm$0.21     96.46$\pm$0.27

AA     88.24$\pm$0.07     88.72$\pm$0.00     88.45$\pm$0.00     63.22$\pm$5.90     84.69$\pm$0.51     83.80$\pm$1.91     84.51$\pm$0.85     89.25$\pm$1.34     86.90$\pm$0.53     82.52$\pm$4.27     87.36$\pm$1.83     92.28$\pm$1.60

k(x100)     93.23$\pm$0.09     89.43$\pm$0.00     86.74$\pm$0.00     48.58$\pm$13.70     89.86$\pm$0.48     89.67$\pm$0.95     90.66$\pm$0.37     94.55$\pm$0.31     92.56$\pm$0.18     89.45$\pm$1.43     92.00$\pm$0.27     95.26$\pm$0.36

The comparative methods mostly misclassify the samples having similar spatial structures (i.e., Meadows and Bare Soil classes for Pavia University dataset) as shown in Table and Figure. Moreover, the overall accuracy for Grapes Untrained is lower than the other classes due to the reasons mentioned above. In a nutshell, one can say that higher accuracy can be achieved by increasing the number of labeled training samples. Thus a higher number of labeled training samples can produce better accuracies for all competing methods.

a) 1PC
b) GT
c) MLR

d) SVM
e) MLP
f) RNN

g) LSTM
h) GRU
i) CNN1D

j) CNN2D
k) CNN3D
l) MorphCNN

Classification Maps obtained by MLR [321], SVM [322], MLP [69], RNN [70], LSTM [323], GRU [324], CNN-1D [218], CNN-2D [325], CNN-3D [326] and MorphCNN [196] on the disjoint train-test dataset for the UT scene.

In general, the works [196, 325] outperformed (i.e. stable results) than the other comparative methods especially in the case of less number of labeled training samples. The above leads to conclude that these works are not sensitive to the number of training samples. Moreover, as the number of training samples increases, the accuracies also increase for these methods however, other methods can work better with a higher number of training samples as compared to these methods. A similar trend has been observed with a higher number of training samples. Thus, one can conclude that the works [196] and [325] can solve the limited availability of training samples issues to some extent while considering disjoint train/test samples.

Classification results obtained by CNN-2D [325] CNN-3D [326], G2C-Conv2D [216], and G2C-Conv3D [216] on the disjoint train-test for IP, PU, Trento, UH, and KSC datasets. The higher accuracies are emphasised.

2*Class     4c||IP        4c||PU      4c||UH      4c||Trento     4cKSC