# Spectral Splitting and Aggregation Network for Hyperspectral Face Super-
Resolution

Junjun Jiang, Chenyang Wang, Kui Jiang, Xianming Liu, and Jiayi Ma  J. Jiang
C. Wang, and X. Liu are with the School of Computer Science and Technology,
Harbin Institute of Technology, Harbin 150001, China, and are also with the
Peng Cheng Laboray, Shenzhen, China. E-mail: {jiangjunjun<EMAIL_ADDRESS>K. Jiang and J. Ma is with the Electronic Information School, Wuhan
University, Wuhan 430072, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

High-resolution (HR) hyperspectral face image plays an important role in face
related computer vision tasks under uncontrolled conditions, such as low-light
environment and spoofing attacks. However, the dense spectral bands of
hyperspectral face images come at the cost of limited amount of photons
reached a narrow spectral window on average, which greatly reduces the spatial
resolution of hyperspectral face images. In this paper, we investigate how to
adapt the deep learning techniques to hyperspectral face image super-
resolution (HFSR), especially when the training samples are very limited.
Benefiting from the amount of spectral bands, in which each band can be seen
as an image, we present a spectral splitting and aggregation network (SSANet)
for HFSR with limited training samples. In the shallow layers, we split the
hyperspectral image into different spectral groups and take each of them as an
individual training sample (in the sense that each group will be fed into the
same network). Then, we gradually aggregate the neighbor bands at the deeper
layers to exploit the spectral correlations. By this spectral splitting and
aggregation strategy (SSAS), we can divide the original hyperspectral image
into multiple samples to support the efficient training of the network and
effectively exploit the spectral correlations among spectrum. To cope with the
challenge of small training sample size (S3) problem, we propose to expand the
training samples by a self-representation model and symmetry-induced
augmentation. Experiments show that the introduced SSANet can well model the
joint correlations of spatial and spectral information. By expanding the
training samples, our proposed method can effectively alleviate the S3
problem. The comparison results demonstrate that our proposed method can
outperform the state-of-the-arts.

###### Index Terms:

Hyperspectral image, face super-resolution, face recognition, self-
representation, spatial-spectral correlation.

##  I Introduction

Benefit from hyperspectral imagery which can capture the local spectral
properties of human tissue, hyperspectral face analysis has attracted more and
more attention from scholars in the field of face related computer vision
tasks because of its satisfactory performance under uncontrolled conditions,
such as low-light environment and spoofing attacks. However, the hyperspectral
face imaging system is often compromised due to the limitations of the amount
of the incident energy. There is always a tradeoff between the spatial and
spectral resolution of the real imaging process. With the increase of the
spectral bands, all other factors kept constant, to ensure a high signal-to-
noise ratio (SNR) the spatial resolution will inevitably become a victim.
Therefore, how to obtain a reliable hyperspectral face image with high spatial
resolution remains a very challenging problem.

Figure 1: The flow chart of our proposed spectral splitting and aggregation
network (SSANet) based hyperspectral face super-resolution (HFSR) method.
SSANet includes three spectral splitting and aggregation modules followed by a
reconstruction module.

Super-Resolution (SR) reconstruction can infer a high-resolution (HR) image
from a single low-resolution (LR) image or sequential observed LR images [1].
It is a post-processing technique that does not require hardware
modifications, and thus could break through the limitations of the imaging
systems. Since the pioneer work of Bake and Kanda [2], face SR, _a.k.i_ , face
hallucination, has received increased attention [3, 4, 5, 6]. Especially in
recent years, the emergence of deep learning technology has promoted a large
number of face SR methods including UR-DGN [7], CBN [8], LCGE [9], Attention-
FH [10], and FSRNet [11]. However, these methods are mainly for gray/RGB face
images and cannot exploit the spectral correlation efficiently. This is mainly
due to that the spectral dimensionality of hyperspectral face image is very
high and the number of training samples of current hyperspectral face image
dataset is extremely small, _e.g._ , around 100. Compared with the general
gray/RGB face datasets, the size of existing hyperspectral face databases is
smaller. For example, the gray/RGB face database has tens of thousands of
samples, while the hyperspectral face database has only a few hundred or even
dozens of face images. And thus, there is not enough training samples to
support the training of the complex deep network. In addition, these deep
learning algorithms require large amounts of memory and are computationally
expensive. Please refer to [12] for more details about deep learning-based
face SR. Whether it is a traditional shallow learning-based method or a deep
learning-based method, it is difficult to construct a powerful representation
model from only dozens of face images to reconstruct the target HR
hyperspectral face images.

In summary, when we apply the existing learning-based face SR methods to
hyperspectral images, the challenges are twofold: (i) the spectral
correlations of hyperspectral face images cannot be fully utilized; (ii)
hyperspectral face images are difficult to obtain, and the training samples of
existing hyperspectral face image database is very small, thus cannot support
these complex learning models.

To this end, in this paper we propose a hyperspectral face super-resolution
(HFSR) method with limited training samples based on a spectral splitting and
aggregation network (SSANet). Thanks to the amount of spectral bands of
hyperspectral data, we split the hyperspectral image into different spectral
groups and take each of them as an individual sample (in the sense that each
group will be fed into the some network). By this splitting strategy, the
training of the complex network can be guaranteed. To exploit the spectral
correlations, which are destroyed by the splitting strategy but very important
for the hyperspectral image reconstruction, we introduce an aggregation
operator to gradually aggregate the neighbor bands at the deeper layers. As
shown in Fig. 1, at the shallow layers, the complex network (with more blocks
or channels) can be well-trained with ‘adequate’ data (each group can be seen
as a sample because of the parameter sharing strategy). At deeper layer, we
design some light structures (with less blocks or channels) because we have
reduce the training samples in order to exploit the spectral correlations. In
general, although we only have a small number of training samples, we can
expect to alleviate the S3 problem through this carefully designed spectral
splitting and aggregation network. Additionally, in order to further cope with
the challenge of S3 problem, we propose to expand the training samples by
self-representation learning. In particular, based on the assumption that
there is a path from the training sample itself to the mean face of all
training samples (as shown in Fig. 2), we develop a training sample expanding
strategy by self-representation learning. Given one training sample, we
leverage the remaining training samples to represent it. According to a
predefined smooth parameter, we can obtain a large number of synthetic
samples, thus forming a path from the training sample itself to the mean face
of the training set. In addition, we also expand the training dataset by the
symmetry of the human face, thus we can directly double the size of the
training dataset.

To sum up, this paper makes the following contributions: (i) We propose the
first SR method for hyperspectral face image. (ii) A novel spectral splitting
and aggregation network is introduced to alleviate the S3 problem, which is
inevitable when we apply deep learning method to the HFSR task. (iii) We
propose some schemes to expand the sample base based on self-representation
and symmetry argumentation.

The rest parts of this study is organized as follows. In Section II, we review
some related methods. Section III presents the details of the proposed method.
In Section IV, we report experimental evaluations of the proposed framework
and compare it with some competitive algorithms. Finally, we conclude this
paper in the last section.

##  II Related Work

Face SR is a very hot topic since the moment it was born. In recent years, a
lot of methods have been proposed. In the following, we will revisit these
approaches from global face based methods to local patch representation based
methods, and then to the deep learning methods that have received much
attention most recently. In addition, we also review some general
hyperspectral image SR methods.

###  II-A Face Model based Super-Resolution

Face model based approaches leverage the global face statistical models, such
as principal component analysis (PCA) [4], locality preserving projection
(LPP) model [13], uniform space projection [14, 15] and nonnegative matrix
factorization (NMF) [16], to model the facial image and super-resolve the
target HR face image globally. They can maintain the main structure of human
face well. However, their reconstruction results lack detailed local facial
features, and inevitably have ghosting artifacts.

###  II-B Patch Representation based Super-Resolution

Considering that the human face is a highly structured subject, many face SR
approaches try to exploit the prior knowledge by dividing the whole face image
into small patches [17, 18, 19]. Among them, position-patch based methods have
gained widespread attention in recent years. However, their solutions for the
linear combination problem are unstable or not unique. To this end, our
previous work [20] introduces the locality-constrained representation (LcR)
technique to simultaneously incorporate sparsity and locality into the patch
representation. With the introduced locality constraint, it obtains a stable
and reasonable representation. In order to alleviate the inconsistency of the
two spaces, LR and HR spaces, some work have been proposed to iteratively
obtain the patch representation and perform neighbor embedding or learn the
mapping in correlation spaces [21, 22, 23, 24, 25, 26].

Figure 2: A vivid illustration of face image “generation”. We assume that all
individual face images (on the big circle) are “evolved” from the mean face
(at the center of the circle). The connection from the center to the circle
represents the evolutionary path, and each point on the evolutionary path can
be seen as the synthesized face images.

###  II-C Deep Learning based Face Super-Resolution

More recently, to reconstruct the latent HR image locally while thinking
globally, DNNs, especially CNNs, have been applied to construct the mapping
relationship between the LR images and their HR counterparts and shown strong
learning capability and accurate prediction of HR images [27, 28]. For
example, Dong _et al._ [27] developed a general image SR method based on
SRCNN. This is the very first attempt to use deep learning tools for image SR
reconstruction. The approach of Liu _et al._ [28] proposes to introduce the
domain expertise to design a Sparse Coding based Network (SCN). Recently,
R-DGN [7], CBN [8], LCGE [9], Attention-FH [10], FSRNet [11], and [29] are the
most competitive approaches for face hallucination. They unitized very deep
networks to model the relationship between the LR images and their HR
counterparts, and verified that deeper networks can produce better results due
to the large receptive field, which means considering more contextual
information, _i.e._ , very large image regions.

###  II-D Deep Learning based Hyperspectral Image Super-Resolution

According to whether an additional guided image (such as panchromatic, RGB, or
multispectral image) is utilized, hyperspectral image SR techniques can be
divided into two categories: multi-source information fusion based
hyperspectral image SR (sometimes called hyperspectral image pansharpening)
and single hyperspectral image SR [30, 31]. The former is to leverage high-
frequency spatial information from HR auxiliary image, and fuse them to the
target HR hyperspectral image [32, 33, 34, 35]. Though achieving good
performance, they need a well co-registered auxiliary image which is arduous
in real applications. Without co-registered auxiliary image, the latter single
hyperspectral image SR methods have still attracted considerable attention.
How to exploit the abundant spectral correlations among successive bands is
the essential problem for them. Traditional methods try to incorporate the
low-rank and group-sparse constraints to exploit the spectral-spatial prior
[36, 37]. Due to its superior performance, deep learning techniques have also
been introduced into the single hyperspectral image SR task. They often adopt
a two-step strategy to get an intermediate result through a deep network
firstly, and then use spectral decomposition constraint to ensure the accuracy
of the reconstructed spectral information [38, 39]. Recently, there are also
some approaches to directly learn an end-to-end deep networks to exploit the
spatial-spectral prior. For example, Mei _et al._ [40] presented a 3D full
convolutional neural network to represent the spectral-spatial information.
Wang _et al_. [41] proposed to combine the advantages of 2D and 3D
convolutions to exploit the spatial and spectral information. In [42], Li _et
al._ developed a grouped deep recursive residual network (GDRRN) based single
hyperspectral image SR method. The designed group-wise convolution and
recursive structure can guarantee that it could yield very good performance.
Inspired by the concept of group convolution, Jiang _et al_. [43] proposed the
spatial-spectral prior network based super-resolution network (SSPSR). Most
recently, inspired by the work of deep image prior [44], the approach of [45]
presents an effective single hyperspectral image restoration algorithm. In
general, these deep methods achieve better results than traditional methods.
However, due to the limited hyperspectral training samples and the high
dimensionality of spectral bands, it is difficult to fully exploit the spatial
information and the correlation among the spectra of the hyperspectral data.

_Summary_ : As for the shallow learning based face SR methods, the local patch
based models have much more strong representation ability than global face
based models, rendering effectively the fine individual details to an input LR
face. To obtain an accurate representation of the image patch, the current
patch representation based methods all try to exploit the image priors
(_e.g_., collaborative, local, sparse, and low-rank constraints) or add some
geometric regularizations from the HR space. They can well deal with the noisy
input, inconsistency between LR and HR manifold spaces. These hand-designed
prior models may not be effective. Deep neural network based methods leverage
large-scale training dataset to train the network, and thus they can obtain
very good performance. However, they may not suitable for our task, where only
about 100 samples are available. Although many SR methods for general
hyperspectral images have been proposed in recent years, they all rely on a
large-scale training set. Therefore, how to adapt the deep learning techniques
for the HFSR task, especially when the training samples are very limited, is
an urgent and extremely challenging issue.

##  III The Proposed Method

Different from gray/RGB face SR problem, in which large paired HR and LR face
training samples can be collected, HFSR can only leverage a small size of
training dataset. Therefore, the very limited data cannot support the
representation and modeling of complex hyperspectral data. To this end, in
this paper we carefully design a deep network based on gradually spectral
splitting and aggregation to alleviate the S3 problem. In addition, we also
propose two strategies to expand the training samples by a self-representation
model and the symmetry-induced augmentation. In this section, we will first
present the details of SSANet and then introduce our training sample expanding
strategies.

Figure 3: RMSE values according to the synthesis controlling parameter. Face
images marked by red, green, and purple are the synthesized training samples,
the HR ground truth face image, and the mean face, respectively. With the
increase of the synthesis controlling parameter, the synthesized training
sample gradually change from the HR ground truth face image to the mean face.

###  III-A Spectral Splitting and Aggregation Network (SSANet)

_Spectral splitting module_ : As we discussed above, because the
dimensionality of hyperspectral images is very high and the training samples
are very limited, it is very difficult for existing deep neural network based
approaches to effectively represent the hyperspectral data. Thanks to the
amount of spectral bands of hyperspectral data, where each spectral band
(neighbor spectral bands) can be seen as a training sample, we split the
hyperspectral image into different groups (samples), thus largely expanding
the size of the original training dataset. Note that we let each spitted group
be fed into a branch network, and let different branches share the same
parameters. Through this parameter sharing based splitting strategy, it will
provide a strong data support for training a deep network.

As shown in Fig. 1, at the shallow layer (_e.g._ , the first spectral
splitting and aggregation stage) we split the hyperspectral data into as many
groups as possible (a band is a group). In this way, we can obtain multiple
training samples, which makes it possible to design a complex network (_i.e._
, many spatial-spectral residual blocks (SSRBs)) to extract features of
hyperspectral data. In the spectral splitting network, each group will be fed
to a branch network to extract the features.

_Aggregation module_ : However, these extracted features may overlook the
correlations among the spectral bands, which are very import to model and
reconstruct the hyperspectral data. Therefore, at the deeper layers, we
gradually increase the size (band number) of the group (_i.e._ , the band
number of each group, from 1 to 4, and then to 8, and finally to all 33 bands
for different spectral splitting and aggregation stages). Meanwhile, we also
let the neighbor groups overlap with each other. When we feed these overlapped
groups to the splitting branch networks, we obtain the output by an
aggregation operator, where the output is averaged according to their spectral
indices. It should be noted that before feeding the outputs of different
branch networks to the following aggregation module, we apply an additional
_Conv_ layer to reduce the channel dimension to the number of input bands.
Thus, each branch network acts like a “reconstruction” network and the input
and output have the same channels. As we know, with the increase of the group
size, the samples available for training will decrease. Thus, at the deeper
layers we design some light structures (with less SSRBs). For example, with
the progress of spectral splitting and aggregation reconstruction, the number
of SSRBs decreases from 12 to 8, 4, and 2 for different spectral splitting and
aggregation stages, respectively.

The proposed SSANet includes three spectral splitting and aggregation modules
followed by a reconstruction module. In order to effectively and efficiently
represent and reconstruct the hyperspectral face images, we carefully set the
numbers of input band as well as the number of SSRB at each spectral splitting
network. In addition, to fully exploit the spectral information of
hyperspectral data, we also change the feature channels of different spectral
splitting networks. At the shallow layer, where the input has only one
spectral band, we set the number of feature channel to 64, and then increase
it to 128, 128, and then to 192. This is because the input has more spectral
bands with the progress of spectral splitting and aggregation reconstruction
(from 1 to 4, 8, and then to 33), it calls for more feature channels to
represent the spectral information of hyperspectral data.

_Upscaling strategy_ : For the proposed SSANet, another design worth
mentioning is the upscaling strategy. The most commonly used upscaling
strategy is the first upsampling (when the input is fed into the network) or
the last upsampling (at the end of the network). In this paper, we are
inspired by the pyramid SR network [46] and propose an upscaling search
strategy. In particular, by inserting upscaling modules at different locations
(before each aggregation module) of the network, we search for a set of
optimized upscaling schemes for different upsampling factors. As shown in Fig.
1, above the input of each splitting network, we give the optimal upscaling
scheme (and the size of the input image) under different upsampling factors,
_e.g._ , $\times 2$, $\times 4$, and $\times 8$. More details can be found at
Section IV-A1.

_Spatial-spectral residual block (SSRB)_ : Different from the commonly used
residual module, we introduce a spatial-spectral residual block [47]. It is a
residual module like block, and includes one $3\times 3$ _Conv_ and one
$1\times 1$ _Conv_ followed by an ReLU layer. We modify the residual module
due to the following considerations. First, the added $1\times 1$ _Conv_ can
be leveraged to reorganize and reweight the importance of spectral bands, thus
efficiently exploiting the spectral correlations of hyperspectral face image.
Second, the added $1\times 1$ _Conv_ can be seen as the purpose of information
distillation and more useful information can be well preserved. Third, it
deepens the depth of the network at a small cost in term of parameters.

Figure 4: The face cube of one example in UWA-HSFD.

###  III-B Training Sample Expanding

As we discussed above, the spectral dimensionality of hyperspectral face image
is very high, but the training sample number of available hyperspectral face
image dataset is extremely small. Therefore, to well represent the high-
dimensional hyperspectral face image, one possible solution is to expand the
training set by synthesizing new training samples.

In this paper, we develop a self-representation learning method to synthesize
the face image. In particular, we assume that all face images are from the
same source, _i.e._ , the mean face, and there is a path from the mean face to
the individual face. As shown in the Fig. 3, the faces on the red line can be
seen as transforming the mean face (top right) to the individual face (bottom
left). Therefore, we can use some states on the path to obtain new samples.
Mathematically, the self-representation based face synthesis method can be
described as following,

$\hat{x}_{i}=\sum_{j=1,j\not=i}^{N}w(x_{i},x_{j})x_{j},$ (1)

where $x_{i}$ is the sample to be reconstructed, $\\{x_{j}|j\not=i,1\leq j\leq
N\\}$ are the training samples except $x_{i}$ itself, and $w(x_{i},x_{j})$
represents the reconstruction weight corresponding to $x_{j}$. From the above
definitions, we can learn that the synthesized training sample is a linear
combination of the training samples except itself. In particular, the
combination weight is proportional to the similarity between current sample
$x_{i}$ and the remaining training samples $x_{j}$,

$w(x_{i},x_{j})=\frac{sim(x_{i},x_{j})}{\sum_{j\not=i,j=1}^{N}sim(x_{i},x_{j})}.$
(2)

Here, $sim(x_{i},x_{j})$ denotes the similarity between $x_{i}$ and $x_{j}$
and is defined as follow,

$sim(x_{i},x_{j})=\exp\left(-\frac{||x_{i}-x_{j}||_{2}^{2}}{\sigma^{2}G}\right),$
(3)

where $\sigma$ denotes the synthesis controlling parameter, and $G$ is the
mean of all elements of $\\{||x_{i}-x_{j}||_{2}^{2}|j\not=i,1\leq j\leq N\\}$.
Since the entire face image is high-dimensional, the globally reconstruction
methods may not accurate or cannot introduce additional information to the
training dataset (please refer to the _Remark 2_). Thus, we decompose the
global face into small image patches, and apply the above-mentioned approach
for each small image patch. Lastly, by integrating all the reconstructed image
patches, we can synthesize a new training sample.

In Fig. 3, it plots the reconstruction error according to the synthesis
controlling parameter $\sigma$, and some synthesized typical face images are
also presented. Through changing the value of $\sigma$, we can obtain
different status (synthesis face samples). The smaller the value of $\sigma$,
the more similar the synthesized face image is to the image to be
reconstructed, and vice versa. When the value of $\sigma$ is very small, these
training samples similar to $x_{i}$ dominate the reconstruction, _i.e._ ,
their reconstruction weights will be large. In contrast, when the value of
$\sigma$ is very large, the reconstruction weights tend to be the same, and
thus obtain the mean face. It is worth noting that when the value of $\sigma$
is very small, the reconstruction weights will be sparse (most of the weights
that associated with these dissimilar training samples will be 0), and the
synthesized face is reconstructed by several similar faces. Due to this
sparsity, we have reasons to believe that the synthesized faces may have some
detailed features (high-frequency information) that do not exist in $x_{i}$.
We explain this phenomenon as follows: a few face samples cannot accurately
represent the original face image $x_{i}$, and the synthetic error is exactly
the high-frequency detail information that does not exist in $x_{i}$. If the
synthesized face image has high frequency detail information that does not
exist in the original face image, we will be able to treat the synthesized
face image as a meaningful sample.

TABLE I: Average quantitative performance by different upsampling strategies over 10 testing images of UWA-HSFD database with respect to 6 PQIs.  up1 | up2 | up3 | up4 | Paras | FLOPs |  CC$\uparrow$ |  SAM$\downarrow$ |  RMSE$\downarrow$ |  ERGAS$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---
1 | 1 | 1 | 2 | 7.285 | 269.2 | 0.9989 | 0.7659 | 0.0052 | 0.4529 | 47.2386 | 0.9919
1 | 1 | 2 | 1 | 6.547 | 296.0 | 0.9988 | 0.7815 | 0.0052 | 0.4599 | 47.0040 | 0.9919
1 | 2 | 1 | 1 | 6.547 | 396.5 | 0.9989 | 0.7555 | 0.0051 | 0.4440 | 47.2672 | 0.9923
2 | 1 | 1 | 1 | 6.104 | 709.9 | 0.9989 | 0.7552 | 0.0051 | 0.4463 | 47.2690 | 0.9922
1 | 1 | 1 | 4 | 8.613 | 74.15 | 0.9954 | 1.3583 | 0.0114 | 0.9632 | 41.0984 | 0.9651
1 | 1 | 2 | 2 | 7.875 | 80.84 | 0.9955 | 1.3352 | 0.0112 | 0.9481 | 41.2323 | 0.9661
1 | 1 | 4 | 1 | 7.137 | 107.6 | 0.9954 | 1.3557 | 0.0114 | 0.9611 | 41.0973 | 0.9656
1 | 2 | 1 | 2 | 7.875 | 106.0 | 0.9957 | 1.2985 | 0.0110 | 0.9269 | 41.4541 | 0.9673
1 | 2 | 2 | 1 | 7.137 | 132.7 | 0.9957 | 1.3168 | 0.0111 | 0.9356 | 41.3544 | 0.9671
1 | 4 | 1 | 1 | 7.137 | 233.2 | 0.9956 | 1.3176 | 0.0112 | 0.9401 | 41.3151 | 0.9669
2 | 1 | 1 | 2 | 7.432 | 184.3 | 0.9957 | 1.2984 | 0.0110 | 0.9274 | 41.4485 | 0.9673
2 | 1 | 2 | 1 | 6.695 | 211.1 | 0.9958 | 1.2890 | 0.0110 | 0.9250 | 41.4659 | 0.9677
2 | 2 | 1 | 1 | 6.695 | 311.5 | 0.9957 | 1.2942 | 0.0110 | 0.9272 | 41.4051 | 0.9676
4 | 1 | 1 | 1 | 6.252 | 625.0 | 0.9956 | 1.2956 | 0.0111 | 0.9342 | 41.3327 | 0.9670
2 | 1 | 1 | 4 | 8.760 | 52.92 | 0.9818 | 2.1048 | 0.0234 | 1.9325 | 35.1264 | 0.8875
1 | 2 | 1 | 4 | 9.203 | 33.33 | 0.9817 | 2.1312 | 0.0235 | 1.9370 | 35.0775 | 0.8856
1 | 1 | 2 | 4 | 9.203 | 27.05 | 0.9821 | 2.1299 | 0.0232 | 1.9146 | 35.1469 | 0.8875
4 | 1 | 1 | 2 | 7.580 | 163.1 | 0.9815 | 2.1160 | 0.0236 | 1.9462 | 35.0292 | 0.8836
1 | 4 | 1 | 2 | 8.465 | 65.14 | 0.9818 | 2.1055 | 0.0233 | 1.9282 | 35.0768 | 0.8868
1 | 1 | 4 | 2 | 8.465 | 33.75 | 0.9815 | 2.1396 | 0.0236 | 1.9461 | 35.0330 | 0.8850
1 | 2 | 2 | 2 | 8.465 | 40.02 | 0.9811 | 2.1280 | 0.0238 | 1.9637 | 34.9534 | 0.8843
2 | 1 | 2 | 2 | 8.023 | 59.61 | 0.9825 | 2.0828 | 0.0230 | 1.8961 | 35.2369 | 0.8901
2 | 2 | 1 | 2 | 8.023 | 84.73 | 0.9824 | 2.0698 | 0.0230 | 1.9011 | 35.2179 | 0.8893
2 | 2 | 2 | 1 | 7.285 | 111.5 | 0.9821 | 2.1056 | 0.0231 | 1.9119 | 35.1661 | 0.8871
8 | 1 | 1 | 1 | 6.400 | 603.7 | 0.9811 | 2.1240 | 0.0238 | 1.9660 | 34.9080 | 0.8827
1 | 8 | 1 | 1 | 7.728 | 192.4 | 0.9817 | 2.1230 | 0.0233 | 1.9293 | 35.0427 | 0.8844
1 | 1 | 8 | 1 | 7.728 | 60.52 | 0.9811 | 2.1995 | 0.0239 | 1.9745 | 34.9172 | 0.8813
1 | 1 | 1 | 8 | 9.940 | 25.38 | 0.9812 | 2.1963 | 0.0238 | 1.9653 | 34.9743 | 0.8816
TABLE II: Effectiveness analysis of the proposed spectral splitting and aggregation structure.  | Dataset | CC$\uparrow$ | SAM$\downarrow$ | RMSE$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---|---|---
w/o SSAS | O | 0.9987 | 0.8100 | 0.0054 | 0.4780 | 46.6650 | 0.9911
O+Self | 0.9990 | 0.7088 | 0.0048 | 0.4179 | 47.9166 | 0.9928
O+Sym | 0.9990 | 0.7271 | 0.0049 | 0.4279 | 47.6540 | 0.9925
O+Self+Sym | 0.9991 | 0.6834 | 0.0046 | 0.4041 | 48.2217 | 0.9932
SSAS | O | 0.9989 | 0.7555 | 0.0051 | 0.4440 | 47.2672 | 0.9923
O+Self | 0.9992 | 0.6552 | 0.0044 | 0.3825 | 48.6323 | 0.9938
O+Sym | 0.9992 | 0.6549 | 0.0043 | 0.3746 | 48.7505 | 0.9941
O+Self+Sym | 0.9993 | 0.6086 | 0.0039 | 0.3430 | 49.4840 | 0.9948
Average Improvements | 0.0002$\uparrow$ | 0.0638$\downarrow$ | 0.0005$\downarrow$ | 0.0460$\downarrow$ | 0.9192$\uparrow$ | 0.0014$\uparrow$

_Remark 1_ : Human face, as a highly structured object, has two eyes, a nose,
and a mouth. The global structure information of different objects is very
similar. The differences between different people are mainly reflected in the
detailed features. Therefore, we have reason to believe that through self-
representation learning, we can synthesize some feature detail information
that is not available in the training dataset. This information can actually
be regarded as a kind of noise compared to the represented face image.
Therefore, the proposed method based on self-representation learning can bring
more information to the training dataset.

_Remark 2_ : It seems that putting a reconstructed sample back to the training
set will add no extra information to the dataset. If we reconstruct the face
image globally, the above point is the truth. But when we reconstruct the face
image locally (patch-wisely), this would be not ture. In other words, for
those patch-based methods, the reconstructed HR result does not have to be a
linear combination of all the original training samples. We can imagine an
extreme situation: if we set the patch size to very small level, _i.e_.,
pixel-wise, we can reconstruct any image (any content) and dose not have to be
a linear combination of training samples, _e.g_., a cat or a dog image.
Therefore, these patch-based reconstruction method can reconstruct some
samples, which dose not have to be a linear combination of the original
training samples. Therefore, when we put reconstructed HR samples back to the
training set, we can actually introduce some additional information in the
sense of that we can add an image (which cannot be linear combination with the
original training) to the training dataset.

##  IV Experiments

We use Pytorch libraries111https://pytorch.org to implement and train the
proposed SSANet method (the code will be available upon acceptance). We train
different models to super-resolve the hyperspectral face images for upsampling
factors 2, 4 and 8 with random initialization. We use the ADAM optimizer [48]
with an initial learning rate of 1e-4 which decays by a factor of 10 at each
30 epochs. In our experiments, we find it will take 60 epochs to achieve the
stable performance. The models are trained with a batch size of 4.

All the experiments are conducted on the UWA Hyperspectral Face Database (UWA-
HSFD)222It should be noted that we only use the database of UWA-HSFD due to
the following reason. Currently, there are only three widely used
hyperspectral face databases, the Hong Kong Polytechnic University
Hyperspectral Face Database (PolyU-HSFD) [49], the CMU Hyperspectral Face
Database (CMU-HSFD) [50], and UWA-HSFD used in our paper. The face images in
PolyU-HSFD are very noisy and not suitable for the supervised learning. As for
CMU-HSFD, the related project is unavailable at present, and we cannot obtain
the database. [51]. It contains 145 hyperspectral face images of 80 subjects
collected in four sessions over time. The face images are acquired by an
indoor imaging system using a CRI’s VariSpec LCTF filter integrated with a
Photon focus camera. In UWA-HSFD [51], the spectral range, in which the images
have been acquired, is from 400 nm to 720 nm with a step size of 10 nm,
resulting 33 spectral bands. Alignment errors are present between individual
bands due to subjects movements and eye blinking during image acquisition. To
simulate the situation that the testing face image is not in the training
phases, we randomly selecting one cube from one session for each of the 70
subjects (includes 10% evaluation samples). The remaining 10 subjects are used
for testing. An example of a face cube from the UWA-HSFD database can be seen
in Fig. 4. In our experiments, all the input LR images are obtained by
2$\times$, 4$\times$, or 8$\times$ Bicubic downsampling.

Evaluation measures. Six widely used quantitative picture quality indices
(PQIs) are employed to evaluate the performance of our method, including cross
correlation (CC) [30], spectral angle mapper (SAM) [52], root mean squared
error (RMSE), erreur relative globale adimensionnelle de synthese (ERGAS)
[53], peak signal-to-noise ratio (PSNR), and structure similarity (SSIM) [54].
For PSNR and SSIM of the reconstructed hyperspectral images, we report their
mean values of all spectral bands. CC, SAM, and ERGAS are three widely adopted
quality indices in HS fusion task, while the remaining three indices are
commonly used quantitative image restoration quality indices. The best values
for these indices are 1, 0, 0, 0, $+\propto$, and 1, respectively.

Figure 5: Reconstructed spectral information of our method with and without
SSAS for four randomly select pixels (they are (40, 55), (110, 113), (20.20),
and (41, 82) from the first testing face image) corresponding to four
subfigures when the upsampling factor is $r$ = 8.

###  IV-A Investigation to the Proposed SSANet

####  IV-A1 Progressively upsampling

The simplest upsampler is to do upsampling at the beginning by Bicubic
interpolation or perform pixel shuffle at the end. They either increase the
parameters and computational complexity of the network or increase the
difficulty of neural network training. Inspired by the Laplacian pyramid SR
network [46], we perform upsampling at the intermediate layers to
progressively upsample the input image. We insert the upsampling module to
each spectral splitting and aggregation module, as shown in Fig. 1. Similar to
the Neural Architecture Search (NAS) [55], in our experiments we manually
search the optimal structure. Table I gives the objective results of some
typical upsampling strategies for different upsampling factors (for $\times$2
and $\times$4, we show all the possible cases, while for $\times$8 we only
show some typical cases). We use “up1”, “up2”, “up3”, and “up4” to denote the
parameters of upsampling factors of different upsampling modules. When the
upsampling factor is 1, we remove the upsampling module. From these results,
we can learn that the upsampling strategy is a very important factor affecting
the overall performance. Different upsampling strategies will not only affect
the reconstruction performance of the network, but also greatly affect the
size and computational complexity of the network. For example, when we insert
the upsampling module at the beginning or at the last for the $\times$2 SR,
the computational complexity of the network is nearly 2.5 times difference
(_e.g._ , 269.2 FLOPs vs. 709.9 FLOPs), despite they achieve the similar
reconstruction performance. The differences of reconstruction performance
under different upsampling strategies of $\times$2, $\times$4, and $\times$8
SR are as high as 0.3 dB in term of PSNR (_e.g._ , 47.0040 dB vs. 47.2690 for
$\times$2 SR, 41.0973 dB vs. 41.4659 for $\times$4 SR, and 34.9080 dB vs.
35.2369 for $\times$2 SR). In order to balance the parameters, model size, and
network reconstruction performance, we choose a reasonable set of upsampling
parameters, which are marked with yellow background.

TABLE III: Effectiveness analysis of the proposed training sample expanding strategy.  Dataset | $r$ | CC$\uparrow$ | SAM$\downarrow$ | RMSE$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---|---|---
O | 2 | 0.9989 | 0.7555 | 0.0051 | 0.4440 | 47.2672 | 0.9923
O+Self | 2 | 0.9992 | 0.6552 | 0.0044 | 0.3825 | 48.6323 | 0.9938
O+Sym | 2 | 0.9992 | 0.6549 | 0.0043 | 0.3746 | 48.7505 | 0.9941
O+Self+Sym | 2 | 0.9993 | 0.6086 | 0.0039 | 0.3430 | 49.4840 | 0.9948
O | 4 | 0.9957 | 1.2985 | 0.0110 | 0.9269 | 41.4541 | 0.9673
O+Self | 4 | 0.9960 | 1.2496 | 0.0107 | 0.9002 | 41.8074 | 0.9687
O+Sym | 4 | 0.9958 | 1.2734 | 0.0109 | 0.9195 | 41.6118 | 0.9678
O+Self+Sym | 4 | 0.9960 | 1.2409 | 0.0107 | 0.9003 | 41.8483 | 0.9689
O | 8 | 0.9825 | 2.0828 | 0.0230 | 1.8961 | 35.2369 | 0.8901
O+Self | 8 | 0.9838 | 1.9939 | 0.0221 | 1.8281 | 35.6496 | 0.8956
O+Sym | 8 | 0.9828 | 2.0371 | 0.0228 | 1.8804 | 35.3614 | 0.8909
O+Self+Sym | 8 | 0.9842 | 1.9934 | 0.0219 | 1.8095 | 35.6852 | 0.8973

####  IV-A2 Spectral splitting and aggregation

To alleviate the S3 problem, we carefully design the spectral splitting and
aggregation strategy (SSAS). It splits the hyperspectral face image into as
many groups as possible at shallow layers to guarantee the training of a
complex network, and gradually aggregates the neighbor bands at deeper layers
to fully exploit spectral correlations. To demonstrate the effectiveness of
SSAS, we report the comparison results of the proposed deep neural network
with and without SSAS at Table II when the upsampling factor is 2. To simulate
the network without SSAS, we maintain the primary structure of SSANet and set
the band size of each branch network to 33 (the number of spectral bands for
UWA-HSFD database). In this situation, there is no spectral splitting and
aggregation process for SSANet. From the tabulated results, we can clearly
learn that SSANet with SSAS can stably improve the performance regardless of
the training datasets (the original training dataset and expanded training
dataset). In all cases, the average improvement (in term of PSNR) of SSANet
with SSAS over SSANet without SSAS is nearly 1.0 dB, and the spectral
similarity has also increased by nearly 10% (please refer to the SAM index).
Fig. 5 plots the reconstructed 33-dimensional spectral vectors (that are
randomly selected from the first testing hyperspectral face image) of the
proposed SSANet with and without SSAS. From these reported results, we can
learn that SSANet with SSAS can better recover the spectral information. This
further verifies that by introducing SSAS, the credibility of the
reconstructed spectrum can be improved.

Figure 6: Face SR results of one object of different methods at bands 5, 10,
15, 20, 25, and 30 when the upsampling factor is 8. (a) HR faces, (b) Bicubic
interpolation, (c) results of LSR [56], (d) results of LcR [20], (e) results
of EDSR [57], (f) results of SAN [58], (g) results of 3DCNN [40], (h) results
of GDRRN [42], (i) results of SSPSR [43], (j) results of Our method.
Figure 7: Face SR results of all 10 objects of different methods at band 30
when the upsampling factor is 8. (a) HR faces, (b) Bicubic interpolation, (c)
results of LSR [56], (d) results of LcR [20], (e) results of EDSR [57], (f)
results of SAN [58], (g) results of 3DCNN [40], (h) results of GDRRN [42], (i)
results of SSPSR [43], (j) results of Our method.
Figure 8: Face SR results of different methods when the upsampling factor is 8. Every three rows give the results of one object of different methods on band 33, 22, and 11. (a) HR faces, (b) results of EDSR [57], (c) results of GDRRN [42], (d) results of SSPSR [43], (e) results of Our method. TABLE IV: Quantitative comparisons of different approaches over 10 testing images from UWA-HSFD dataset with respect to six PQIs. The best and second best results are highlighted in boldface and underlined, respectively. | $r$ | CC$\uparrow$ | SAM$\downarrow$ | RMSE$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---|---|---
LSR [56] | 2 | 0.9977 | 1.2394 | 0.0076 | 0.6526 | 43.7839 | 0.9784
LcR [20] | 2 | 0.9980 | 1.0751 | 0.0073 | 0.6214 | 44.4102 | 0.9808
EDSR [57] | 2 | 0.9988 | 0.7777 | 0.0053 | 0.4616 | 46.9760 | 0.9917
SAN [58] | 2 | 0.9985 | 0.8484 | 0.0055 | 0.4921 | 46.3874 | 0.9906
3DCNN [40] | 2 | 0.9986 | 0.8676 | 0.0061 | 0.5218 | 46.2634 | 0.9895
GDRRN [42] | 2 | 0.9976 | 0.7848 | 0.0054 | 0.4730 | 46.8245 | 0.9911
SSPSR [43] | 2 | 0.9987 | 0.7947 | 0.0052 | 0.4614 | 46.9406 | 0.9916
SSANet | 2 | 0.9989 | 0.7555 | 0.0051 | 0.4440 | 47.2672 | 0.9923
LSR [56] | 4 | 0.9937 | 1.6073 | 0.0134 | 1.1214 | 39.6160 | 0.9472
LcR [20] | 4 | 0.9939 | 1.5774 | 0.0134 | 1.1144 | 39.7621 | 0.9480
EDSR [57] | 4 | 0.9949 | 1.4242 | 0.0118 | 0.9995 | 40.6932 | 0.9628
SAN [58] | 4 | 0.9948 | 1.4723 | 0.0118 | 1.0048 | 40.6052 | 0.9629
3DCNN [40] | 4 | 0.9942 | 1.4482 | 0.0132 | 1.0984 | 40.0966 | 0.9556
GDRRN [42] | 4 | 0.9950 | 1.3897 | 0.0118 | 0.9939 | 40.7684 | 0.9626
SSPSR [43] | 4 | 0.9953 | 1.3750 | 0.0114 | 0.9656 | 41.0340 | 0.9651
SSANet | 4 | 0.9957 | 1.2985 | 0.0110 | 0.9269 | 41.4541 | 0.9673
LSR [56] | 8 | 0.9795 | 2.3873 | 0.0246 | 2.0349 | 34.4513 | 0.8782
LcR [20] | 8 | 0.9810 | 2.3680 | 0.0240 | 1.9775 | 34.8670 | 0.8835
EDSR [57] | 8 | 0.9820 | 2.2191 | 0.0232 | 1.9213 | 35.1152 | 0.8849
SAN [58] | 8 | 0.9815 | 2.3889 | 0.0234 | 1.9420 | 34.9323 | 0.8855
3DCNN [40] | 8 | 0.9741 | 2.2846 | 0.0283 | 2.3150 | 33.5437 | 0.8470
GDRRN [42] | 8 | 0.9785 | 2.1964 | 0.0254 | 2.0910 | 34.3277 | 0.8704
SSPSR [43] | 8 | 0.9816 | 2.1979 | 0.0234 | 1.9356 | 34.9966 | 0.8856
SSANet | 8 | 0.9825 | 2.0828 | 0.0230 | 1.8961 | 35.2369 | 0.8901

###  IV-B Effectiveness of Training Sample Expanding

In this paper, we introduce two strategies to expand the size of training
dataset. To demonstrate the effectiveness of these strategies, we show the
comparison results of our method with different training datasets. As shown in
Table III, “O” demotes the original training dataset, “Self” demotes the
expanded training dataset with self-representation based data argumentation,
and “Sym” is the expanded training dataset using the symmetry nature of humane
to flip the face image to double the size of training dataset. From the
comparison results, we can clearly learn that the proposed self-representation
and symmetry based data argumentation methods can improve the performance of
our proposed deep neural network separately, no matter what the upsampling
factor $r$ is. The combination of these two strategies can achieve the best
performance.

###  IV-C Comparison Results

Since the proposed SSANet is the first hyperspectral image SR method for human
faces, there is no direct hyperspectral image SR method for comparison.
Therefore, to demonstrate the effectiveness of our method, we adjust two
traditional shallow learning based face SR methods, LSR [56] and LcR [20], two
deep learning based general image SR methods, EDSR [57] and SAN [58], and one
most recently proposed deep learning based single hyperspectral image SR
method, SSPSR [43] for the hyperspectral image SR task. Specifically, as for
LSR [56] and LcR [20], we super-resolve each band of the hyperspectral face
image separately. For EDSR [57] and SAN [58], we retrain their networks by
changing the number of input channels with the UWA-HSFD database. SSPSR [43]
is also retrained by the UWA-HSFD database for fair comparison.

Fig. 6 shows the super-resolved results of one object of different SR
approaches at five specific bands, and in Fig. 7 we also present the results
of all objects at band 30. From these comparison results, we learn that
Bicubic interpolation lost most of the face detailed features. LSR [56] and
LcR [20] require the accurate alignment of face images. However, face images
cannot be completely aligned due to non-rigid transformations such as
expressions and poses. Therefore, ghosting effects often appear at the regions
where the eyes and mouth cannot be completely aligned. EDSR [57] and SAN [58]
are not specifically designed for hyperspectral faces, and they cannot fully
exploit the rich spectral information of hyperspectral faces. Their results
are much blur than SSPSR [43], which is a general hyperspectral image SR
method. Results reconstructed by our proposed method are more credible (more
similar to the ground truth), and at the same time our method greatly reduces
the artificial effects of the reconstructed eyes, mouth, and nose.

It should be noted that almost all these comparison approaches cannot
accurately reconstruct the slightly opened eyes. As shown in Fig. 8, the eyes
of the input face image are slightly opened, but the reconstruction result are
open (please refer to the first row of the first object and the results of the
second object). Actually, this shows a weakness of these learning-based
techniques which depend highly upon the consistency (similarity) between the
supporting testing image and training dataset. This is mainly because the
training data cannot cover the changes caused by all factors (such as posture,
expression, lighting, _etc_.), especially when the size of the training
dataset is limited.

Table IV tabulates the average objective results of seven comparison methods
on the 10 testing images from UWA-HSFD dataset when the upsampling factor is
set to 2, 4, and 8. Note that SSANet is the model trained by the original
training samples, while SSANet+ denotes the model trained by the expanded
training samples with “Self” and “Sym”. These objective results also
demonstrate the better performance of our method.

Figure 9: Reconstructed images of _paints_ at CAVE dataset with upsampling
factor $r$ = 4. The first row is the reconstructed results of the comparison
methods, while the last row is the corresponding error maps. At the bottom of
each reconstructed result, we also report the PSNR (dB) and SSIM indices.
Figure 10: Reconstructed images of _real_and_fake_apples_ at CAVE dataset with
upsampling factor $r$ = 4. The first row is the reconstructed results of the
comparison methods, while the last row is the corresponding error maps. At the
bottom of each reconstructed result, we also report the PSNR (dB) and SSIM
indices.

###  IV-D Results on CAVE dataset

Since UWA-HSFD is the only released and usable hyperspectral face image
database, we just show the face SR comparison results on that database. In
this section, to demonstrate the effectiveness of the proposed spectral
splitting and aggregation network, we additionally conduct comparison
experiments on hyperspectral images of natural scenes. Specifically, we use
the CAVE multispectral image database because it is widely used in many
multispectral image recovery tasks. The database consists of 32 scenes of
everyday objects with spatial size of 512$\times$512, including 31 spectral
bands ranging from 400nm to 700nm at 10nm steps. To prepare samples for
training, we randomly select 20 hyperspectral images from the database (10%
samples are randomly selected for evaluations). When the upsampling factor $r$
is 4, we extract patches with 64$\times$64 pixels (32 pixels overlap) for
training; when the upsampling factor $r$ is 8, we let the extracted patches be
128$\times$128 pixels (with 64 pixels overlap). The corresponding LR
hyperspectral images are generated by Bicubic downsampling with a factor of 4
or 8. The remaining 12 hyperspectral images of the database are used for
testing, where the original images are treated as ground truth HR
hyperspectral images, and the LR hyperspectral inputs are generated similarly
as the training samples.

We compare the proposed SSANet method with some very competitive approaches,
EDSR [57], RCAN [59], SAN [58], 3DCNN [40], GDRRN [42], and two most recently
proposed methods, SSPSR [43] and MCNet [41]. The average performance of the
CC, SAM, RMSE, ERGAS, PSNR, and SSIM results of competing methods for
different upsampling factors on the CAVE dataset is reported in Table V. From
these results, we notice that SSPSR [43], MCNet [41], and the proposed SSANet
method outperform other competing methods with large gains. The proposed
SSANet method is slightly better than SSPSR [43] and MCNet [41], which are
state-of-the-arts in the field of single hyperspectral image SR.

In Fig. 9 and Fig. 10, we show the super-resolved visual results (pseudo RGB
image) and the corresponding error maps at 480nm, 580nm and 680nm of different
methods for testing images _paints_ and _real_and_fake_apples_ with upsampling
factors $r$ = 4, respectively. From these visual results, we learn that all
the competing methods can well recover the HR spatial structures of the
hyperspectral images. From these error maps, we learn that RCAN, SSPSR, MCNet,
and the proposed method achieve better reconstruction fidelity. Below the
reconstruction results, we also present the PSNR, and SSIM results of the
pseudo RGB image for different comparison methods. Obviously, the proposed
SSANet method performs best in most cases, which is consistent with the
results in Table V.

TABLE V: Quantitative comparisons of different approaches over 12 testing images from CAVE dataset with respect to six PQIs. The best and second best results are highlighted in boldface and underlined, respectively. | $r$ | CC$\uparrow$ | SAM$\downarrow$ | RMSE$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
---|---|---|---|---|---|---|---
Bicubic | 4 | 0.9868 | 4.1759 | 0.0212 | 5.2719 | 34.7214 | 0.9277
EDSR [57] | 4 | 0.9931 | 3.5499 | 0.0149 | 3.5921 | 38.1575 | 0.9522
RCAN [59] | 4 | 0.9935 | 3.6050 | 0.0142 | 3.4178 | 38.7585 | 0.9530
SAN [58] | 4 | 0.9935 | 3.5951 | 0.0143 | 3.4200 | 38.7188 | 0.9531
3DCNN [40] | 4 | 0.9928 | 3.3463 | 0.0154 | 3.7042 | 37.9759 | 0.9522
GDRRN [42] | 4 | 0.9934 | 3.4143 | 0.0145 | 3.5086 | 38.4507 | 0.9538
SSPSR [43] | 4 | 0.9939 | 3.1846 | 0.0138 | 3.3384 | 39.0892 | 0.9553
MCNet [41] | 4 | 0.9942 | 3.1758 | 0.0135 | 3.2767 | 39.1059 | 0.9605
SSANet | 4 | 0.9942 | 3.2137 | 0.0134 | 3.1976 | 39.3539 | 0.9596
Bicubic | 8 | 0.9666 | 5.8962 | 0.0346 | 4.2175 | 30.2056 | 0.8526
EDSR [57] | 8 | 0.9778 | 5.6865 | 0.0279 | 3.3903 | 32.4072 | 0.8842
RCAN [59] | 8 | 0.9791 | 5.9771 | 0.0268 | 3.1781 | 32.9544 | 0.8884
SAN [58] | 8 | 0.9795 | 5.8683 | 0.0267 | 3.1437 | 33.0012 | 0.8888
3DCNN [40] | 8 | 0.9755 | 5.0948 | 0.0292 | 3.5536 | 31.9691 | 0.8863
GDRRN [42] | 8 | 0.9769 | 5.3597 | 0.0280 | 3.3460 | 32.5763 | 0.8890
SSPSR [43] | 8 | 0.9805 | 4.4874 | 0.0257 | 3.0419 | 33.4340 | 0.9010
MCNet [41] | 8 | 0.9813 | 4.6726 | 0.0248 | 2.9006 | 33.8149 | 0.9031
SSANet | 8 | 0.9825 | 4.3583 | 0.0241 | 2.8988 | 34.0915 | 0.9049

##  V Conclusions

In this paper, we present a hyperspectral face super-resolution (HFSR)
approach based on spectral splitting and aggregation network (SSANet). This is
the first face SR work focusing on hyperspectral images. Different from
traditional face and general image SR tasks, in which there are enough
training samples to support the training of a complex network, HFSR has to
face the small training sample size (S3) problem. To this end, on the one hand
we carefully design a spectral splitting and aggregation network to fully make
use of multiple spectral information. On the other hand we introduce two
strategies to expand the training samples by a self-representation and the
symmetry augmentation. Experimental results on public hyperspectral face
database demonstrate that our proposed SSANet method and the self-
representation and symmetry based training sample expanding strategy are
effective for the HFSR task. In addition, we also report the comparison
results on hyperspectral images of natural scenes, which demonstrate the
generalization of our method.

## References

  * [1] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” _IEEE Signal Process. Magazine_ , vol. 20, no. 3, pp. 21–36, may 2003.
  * [2] S. Baker and T. Kanade, “Hallucinating faces,” in _FG_ , 2000, pp. 83–88.
  * [3] C. Liu, H.-Y. Shum, and C.-S. Zhang, “A two-step approach to hallucinating faces: global parametric model and local nonparametric model,” in _CVPR_ , vol. 1, 2001, pp. 192–198.
  * [4] X. Wang and X. Tang, “Hallucinating face by eigentransformation,” _IEEE Trans. Syst. Man Cybern. Part C-Appl. Rev._ , vol. 35, no. 3, pp. 425–434, 2005\.
  * [5] Y. Zhuang, J. Zhang, and F. Wu, “Hallucinating faces: Lph super-resolution and neighbor reconstruction for residue compensation,” _Pattern Recogn._ , vol. 40, no. 11, pp. 3178–3194, 2007.
  * [6] J.-S. Park and S.-W. Lee, “An example-based face hallucination method for single-frame, low-resolution facial images,” _IEEE Trans. Image Process._ , vol. 17, no. 10, pp. 1806–1816, Oct 2008.
  * [7] X. Yu and F. Porikli, “Ultra-resolving face images by discriminative generative networks,” in _ECCV_. Springer, 2016, pp. 318–333.
  * [8] S. Zhu, S. Liu, C. C. Loy, and X. Tang, “Deep cascaded bi-network for face hallucination,” in _ECCV_. Springer, 2016, pp. 614–630.
  * [9] Y. Song, J. Zhang, S. He, L. Bao, and Q. Yang, “Learning to hallucinate face images via component generation and enhancement,” in _IJCAI_ , 2017, pp. 4537–4543.
  * [10] Q. Cao, L. Lin, Y. Shi, X. Liang, and G. Li, “Attention-aware face hallucination via deep reinforcement learning,” in _CVPR_ , 2017, pp. 1656–1664.
  * [11] Y. Chen, Y. Tai, X. Liu, C. Shen, and J. Yang, “Fsrnet: End-to-end learning face super-resolution with facial priors,” in _CVPR_ , 2008, pp. 1–8.
  * [12] J. Jiang, C. Wang, X. Liu, and J. Ma, “Deep learning-based face super-resolution: A survey,” _arXiv preprint arXiv:2101.03749_ , 2021.
  * [13] S. W. Park and M. Savvides, “Breaking the limitation of manifold analysis for super-resolution of facial images,” in _ICASSP_ , vol. 1, Apr 2007, pp. I–573–I–576.
  * [14] H. Huang, H. He, X. Fan, and J. Zhang, “Super-resolution of human face image using canonical correlation analysis,” _Pattern Recogn._ , vol. 43, no. 7, pp. 2532–2543, 2010.
  * [15] L. An and B. Bhanu, “Face image super-resolution using 2D CCA,” _Signal Proc_ , vol. 103, pp. 184–194, 2014.
  * [16] J. Yang, J. Wright, T. Huang, , and Y. Ma, “Image super-resolution via sparse representation,” _IEEE Trans. Image Process._ , vol. 19, no. 11, pp. 2861–2873, 2010.
  * [17] C.-Y. Yang, S. Liu, and M.-H. Yang, “Structured face hallucination,” in _CVPR_ , 2013, pp. 1099–1106.
  * [18] Y. Li, C. Cai, G. Qiu, and K.-M. Lam, “Face hallucination based on sparse local-pixel structure,” _Pattern Recogn._ , vol. 47, no. 3, pp. 1261–1270, 2014.
  * [19] Z. Hui, W. Liu, and K.-M. Lam, “A novel correspondence-based face-hallucination method,” _Image and Vision Computing_ , 2017.
  * [20] J. Jiang, R. Hu, Z. Wang, and Z. Han, “Noise robust face hallucination via locality-constrained representation,” _IEEE Trans. Multimedia_ , vol. 16, no. 5, pp. 1268–1281, Aug 2014.
  * [21] Q. Zhang, F. Zhou, F. Yang, and Q. Liao, “Face super-resolution via semi-kernel partial least squares and dictionaries coding,” in _DSP 2015_ , 2015, pp. 590–594.
  * [22] R. A. Farrugia and C. Guillemot, “Face hallucination using linear models of coupled sparse support,” _IEEE Trans. Image Process._ , vol. 26, no. 9, pp. 4562–4577, Sept 2017.
  * [23] G. Gao, Y. Yu, J. Xie, J. Yang, M. Yang, and J. Zhang, “Constructing multilayer locality-constrained matrix regression framework for noise robust face super-resolution,” _Pattern Recognition_ , p. 107539, 2020.
  * [24] L. Liu, C. L. P. Chen, S. Li, Y. Y. Tang, and L. Chen, “Robust face hallucination via locality-constrained bi-layer representation,” _IEEE Trans. Cybern._ , vol. 48, no. 4, pp. 1189–1201, 2018.
  * [25] J. Shi, X. Liu, Y. Zong, C. Qi, and G. Zhao, “Hallucinating face image by regularization models in high-resolution feature space,” _IEEE Trans. Image Process._ , vol. 27, no. 6, pp. 2980–2995, 2018.
  * [26] L. Chen, J. Pan, J. Jiang, J. Zhang, and Y. Wu, “Robust face super-resolution via position relation model based on global face context,” _IEEE Trans. Image Process._ , vol. 29, pp. 9002–9016, 2020.
  * [27] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 38, no. 2, pp. 295–307, 2016.
  * [28] D. Liu, Z. Wang, B. Wen, J. Yang, W. Han, and T. S. Huang, “Robust single image super-resolution via deep networks with sparse prior,” _IEEE Trans. Image Process._ , vol. 25, no. 7, pp. 3194–3207, 2016.
  * [29] X. Yu and F. Porikli, “Imagining the unimaginable faces by deconvolutional networks,” _IEEE Trans. Image Process._ , vol. 27, no. 6, pp. 2747–2761, June 2018.
  * [30] L. Loncan, L. B. De Almeida, J. M. Bioucas-Dias, X. Briottet, J. Chanussot, N. Dobigeon, S. Fabre, W. Liao, G. A. Licciardi, M. Simoes _et al._ , “Hyperspectral pansharpening: A review,” _IEEE Geoscience and remote sensing magazine_ , vol. 3, no. 3, pp. 27–46, 2015.
  * [31] N. Yokoya, C. Grohnfeldt, and J. Chanussot, “Hyperspectral and multispectral data fusion: A comparative review of the recent literature,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 5, no. 2, pp. 29–56, 2017.
  * [32] N. Yokoya, T. Yairi, and A. Iwasaki, “Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 50, no. 2, pp. 528–537, 2011\.
  * [33] W. Dong, F. Fu, G. Shi, X. Cao, J. Wu, G. Li, and X. Li, “Hyperspectral image super-resolution via non-negative structured sparse representation,” _IEEE Trans. Image Process._ , vol. 25, no. 5, pp. 2337–2352, 2016.
  * [34] Q. Xie, M. Zhou, Q. Zhao, D. Meng, W. Zuo, and Z. Xu, “Multispectral and hyperspectral image fusion by ms/hs fusion net,” in _CVPR_ , 2019, pp. 1585–1594.
  * [35] R. A. Borsoi, T. Imbiriba, and J. C. M. Bermudez, “Super-resolution for hyperspectral and multispectral image fusion accounting for seasonal spectral variability,” _IEEE Trans. Image Process._ , vol. 29, pp. 116–127, 2020\.
  * [36] H. Huang, A. G. Christodoulou, and W. Sun, “Super-resolution hyperspectral imaging with unknown blurring by low-rank and group-sparse modeling,” in _ICIP_. IEEE, 2014, pp. 2155–2159.
  * [37] J. Li, Q. Yuan, H. Shen, X. Meng, and L. Zhang, “Hyperspectral image super-resolution by spectral mixture analysis and spatial–spectral group sparsity,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 13, no. 9, pp. 1250–1254, 2016.
  * [38] Y. Yuan, X. Zheng, and X. Lu, “Hyperspectral image superresolution by transfer learning,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 10, no. 5, pp. 1963–1974, 2017.
  * [39] W. Xie, X. Jia, Y. Li, and J. Lei, “Hyperspectral image super-resolution using deep feature matrix factorization,” _IEEE Trans. Geosci. Remote Sens._ , vol. 57, no. 8, pp. 6055–6067, 2019.
  * [40] S. Mei, X. Yuan, J. Ji, Y. Zhang, S. Wan, and Q. Du, “Hyperspectral image spatial super-resolution via 3d full convolutional neural network,” _Remote Sensing_ , vol. 9, no. 11, p. 1139, 2017.
  * [41] Q. Li, Q. Wang, and X. Li, “Mixed 2d/3d convolutional network for hyperspectral image super-resolution,” _Remote Sensing_ , vol. 12, no. 10, p. 1660, 2020.
  * [42] Y. Li, L. Zhang, C. Dingl, W. Wei, and Y. Zhang, “Single hyperspectral image super-resolution with grouped deep recursive residual network,” in _BigMM_. IEEE, 2018, pp. 1–4.
  * [43] J. Jiang, H. Sun, X. Liu, and J. Ma, “Learning spatial-spectral prior for super-resolution of hyperspectral imagery,” _IEEE Trans. Computat. Imag._ , vol. 6, pp. 1082–1096, 2020.
  * [44] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” in _CVPR_ , 2018, pp. 9446–9454.
  * [45] O. Sidorov and J. Y. Hardeberg, “Deep hyperspectral prior: Single-image denoising, inpainting, super-resolution,” in _ICCVW_ , 2019, pp. 3844–3851.
  * [46] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” in _CVPR_ , 2017, pp. 624–632.
  * [47] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [48] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [49] W. Di, L. Zhang, D. Zhang, and Q. Pan, “Studies on hyperspectral face recognition in visible spectrum with feature band selection,” _IEEE Trans. Syst., Man, and Cybern. A: Sys. and Humans_ , vol. 40, no. 6, pp. 1354–1361, 2010.
  * [50] L. J. Denes, P. Metes, and Y. Liu, “Hyperspectral face database,” Carnegie Mellon University, Pittsburgh, PA, Tech. Rep. CMU-RI-TR-02-25, September 2002\.
  * [51] M. Uzair, A. Mahmood, and A. Mian, “Hyperspectral face recognition with spatiospectral information fusion and pls regression,” _IEEE Trans. Image Process._ , vol. 24, no. 3, pp. 1127–1137, 2015.
  * [52] R. H. Yuhas, A. F. Goetz, and J. W. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm,” in _JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, pp. 147-149_ , 1992.
  * [53] L. Wald, _Data fusion: definitions and architectures: fusion of images of different spatial resolutions_. Presses des MINES, 2002.
  * [54] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli _et al._ , “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600–612, 2004.
  * [55] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in _ECCV_ , 2018, pp. 19–34.
  * [56] X. Ma, J. Zhang, and C. Qi, “Hallucinating face by position-patch,” _Pattern Recogn._ , vol. 43, no. 6, pp. 2224–2236, 2010.
  * [57] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in _CVPRW_ , 2017, pp. 136–144.
  * [58] T. Dai, J. Cai, Y. Zhang, S.-T. Xia, and L. Zhang, “Second-order attention network for single image super-resolution,” in _CVPR_ , 2019, pp. 11 065–11 074.
  * [59] Y. Zhang, K. Li, K. Li, L. Wang, B. Zhong, and Y. Fu, “Image super-resolution using very deep residual channel attention networks,” in _ECCV_ , 2018, pp. 286–301.