# Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-
Resolution

Qing Ma1 Junjun Jiang1,2 Xianming Liu1,2 Jiayi Ma3
1Harbin Institute of Technology 2Peng Cheng Laboratory 3Wuhan University
Corresponding author<EMAIL_ADDRESS>
###### Abstract

To solve the ill-posed problem of hyperspectral image super-resolution
(HSISR), an usually method is to use the prior information of the
hyperspectral images (HSIs) as a regularization term to constrain the
objective function. Model-based methods using hand-crafted priors cannot fully
characterize the properties of HSIs. Learning-based methods usually use a
convolutional neural network (CNN) to learn the implicit priors of HSIs.
However, the learning ability of CNN is limited, it only considers the spatial
characteristics of the HSIs and ignores the spectral characteristics, and
convolution is not effective for long-range dependency modeling. There is
still a lot of room for improvement. In this paper, we propose a novel HSISR
method that uses Transformer instead of CNN to learn the prior of HSIs.
Specifically, we first use the proximal gradient algorithm to solve the HSISR
model, and then use an unfolding network to simulate the iterative solution
processes. The self-attention layer of Transformer makes it have the ability
of spatial global interaction. In addition, we add 3D-CNN behind the
Transformer layers to better explore the spatio-spectral correlation of HSIs.
Both quantitative and visual results on two widely used HSI datasets and the
real-world dataset demonstrate that the proposed method achieves a
considerable gain compared to all the mainstream algorithms including the most
competitive conventional methods and the recently proposed deep learning-based
methods.

##  1 Introduction

Hyperspectral images (HSIs) have high spectral resolution and are widely used
in various computer vision tasks, including target recognition and tracking
[28, 33], medical image processing [1], and remote sensing [17, 6]. While the
HSI can achieve a high spectral resolution with contiguous and narrow bands,
its spatial resolution is usually much coarser than that of the RGB images in
our daily life. This is due to the fact that the dense spectral bands in the
hyperspectral sensors make a limited amount of photons reached one narrow
spectral window averagely. The low spatial resolution of the HSI captured by
the sensor greatly limits its value for application. A natural solution is to
instead capture a low-resolution (LR) HSI and a high-resolution (HR)
multispectral image (MSI), _e.g._ , RGB image, and fuse them into a resultant
image with high spatial and spectral resolution simultaneously. This procedure
is referred to as hyperspectral multispectral image fusion (HS/MS fusion) or
hyperspectral super-resolution.

Traditional HS/MS fusion methods employ prior knowledge of HSIs as regularizer
to solve such a seriously ill-posed problem, _e.g._ , sparse representation
[2, 21, 12], low-rank prior [35, 51], non-local [8, 52, 44] and self-
similarity [19]. They have achieved better and better performance, because
these priors are getting closer and closer to the essential characteristics of
the data. However, the prior knowledge is predefined manually and is born with
the following shortcomings. On the one hand, a high level of wisdom is
required. On the other hand, these hand-crafted priors have limitations and
often fail to reflect all the characteristics of the data. These severely
restrict the performance and generalization ability of the optimization-based
methods mentioned above.

Given the recent advances in deep learning, many convolutional neural networks
(CNNs) based image super-resolution approaches have garnered attention
recently. To solve the above problems of traditional HS/MS fusion methods,
some researchers use deep learning to cope with the fusion based HSISR
problem. Unlike traditional methods of manually designing a prior, these deep
learning-based methods embrace a large amount of data to learn data-driven
priors, greatly improving the fusion performance. Researchers have proposed
some specialized modules, such as spatial attentions, channel attentions, and
joint ones, to exploit the spatial, spectral, or spatio-spectral priors of
HSIs. To take advantage of the observation model and address the gap between
the optimization-based and learning-based methods, most recently, a series of
HS/MS fusion methods based on unfolding networks have been proposed. For
example, Xie _et al._ [43] proposed an effective unfolding network based on
MHF-net, which first constructs an MS/HS fusion model and then builds the
proposed network by unfolding the proximal gradient algorithm to solve the
proposed model. Wang _et al._ [37] proposed the DBIN model, where the
estimation of the observation model and the fusion process is optimized
iteratively and alternatively during the super-resolution reconstruction. Dong
_et al._ [14] proposed an approach based on MoG-DCN, an iterative HSISR
algorithm based on a deep HSI denoiser to leverage both domain knowledge
likelihood and deep image prior. Essentially, these methods all try to cast
the HS/MS fusion optimization problem into joint learning of the observation
model and a deep denoiser prior. The intrinsic difference among the above
three tasks lies in how to model the denoiser prior. MHF-net and DBIN stack
several ResBlocks [20] to model the data prior, while MoG-DCN leverages a
relatively complex U-net [30] to exploit the prior.

In this paper, we try to design a more powerful network that can fully extract
the priors of HSIs. We observe that the HSIs are three-dimensional data cubes,
and their priors can be divided into two aspects: spatial priors and spectral
priors. We need to design appropriate networks for these two aspects to better
learn the priors of HSIs. We propose to use Transformer [34] to learn the
spatial prior of HSIs. The self-attention mechanism in the Transformer
requires each pixel of the image to pay attention to each other pixel, so that
Transformer has long-range modeling capabilities and has achieved good
performance in many visual tasks. It is very suitable for extracting the
spatial prior of HSIs. As for the spectral prior, 3D-CNN is a natural and
effective choice. It can pay attention to the correlation of spatio-spectra
while learning spectral priors. Keeping these in mind, we propose an HS/MS
fusion model, namely 3D-CNN and Transformer prior network (3DT-Net). We first
use the proximal gradient algorithm to solve the optimization problem with the
observation models of LR-HSIs and HR-MSIs, and then we unfold the iterative
process of the proximal gradient algorithm into a multistage network.
Particularly, in each iteration, unlike previous learning-based methods that
construct 2D-CNN to learn priors, we design a 3D-CNN and Transformer network
with Swin Transformer layers [25] to exploit spatial priors and 3D
convolutional layers to exploit spatio-spectral correlation priors.
Experimental results have shown that the proposed 3DT-Net outperforms many
recently proposed HS/MS fusion methods.

Figure 1: The flowchart of our proposed 3DT-Net deep unrolling framework.

##  2 Related Work

In this section, we introduce related existing methods, particularly focusing
prior modeling in hyperspectral image super-resolution and vision Transformer.

###  2.1 Prior Modeling in HSISR

HSIs super-resolution is an ill-posed problem, a common solution is to choose
a prior or regularization to constrain the optimization equation. Various
priors have been already exploited to regularize the HSIs Super-resolution
problem. In [27], Frosti _et al._ used total variation to regularize an ill-
posed problem dictated by a widely used explicit image formation model.
Considering that HSIs are redundant in nature, sparse prior can be used to
constrain the spatial and spectral correlation of HSIs. Wei _et al._ [40]
designed a sparse regularization term, relying on a decomposition of the scene
on a set of dictionaries. In [2], Akhtar _et al._ proposed a generic Bayesian
sparse coding strategy to be used with Bayesian dictionaries learned with the
Beta process. In addition, spectral unmixing prior [47, 41] and self-
similarity [19] are also often used to address HSISR. The main drawback of
these methods is the quality of the recovered HSIs highly depend on whether
the selected prior is appropriate. In addition, it requires manual tweaking of
its balancing parameters between the data term and prior term.

In recent years, deep learning have pushed forward the frontier of many
computer vision tasks, including image classification and object detection
[20, 18], image retrieval [29], and so on. Inspired by these representative
work, using deep learning methods to accomplish the HS/MS fusion has attracted
wide attention. Different from the traditional method of manual design a
prior, the deep learning-based methods use a large amount of data to learn
data-driven priors. Dian _et al._ [10] combined traditional methods with deep
learning methods. They first initialize the HR-HSI by solving a Sylvester
equation, and then use deep CNN to learn the priors of the target HR-HSI. Xie
_et al._ [43] and Wu Wang _et al._ [37] used ResNet to learn HSIs priors. Dian
_et al._ [11] proposed using a CNN denoiser to regularize the HS/MS fusion
model. In [14], Dong _et al._ proposed using U-net to replace ResNet to learn
a denoising prior for HS/MS fusion.

###  2.2 Vision Transformer

Transformer was proposed by Vaswani _et al._ [34] for NLP. Since directly
replacing the CNN convolution layers with Transformer layers to process images
will bring a huge computational burden, the early methods are to augment a
standard CNN model with Transformer layers [5, 32, 38, 46]. Recently, Chen _et
al._ [7] proposed a backbone model IPT for image restoration based on
Transformer. However, IPT has a huge amount of parameters and calculations,
and it is very difficult to train such a model. In order to solve the problem
of large amount of calculation when using the Transformer to process image
tasks, Dosovitskiy _et al._ [15] proposed to split the training image into a
sequence of image patches. However, [15] cannot handle high-resolution images,
due to its low-resolution feature maps and the quadratic increase in
complexity with image size. Follow up [15], Liu _et al._ proposed Swin
Transformer [25], a hierarchical Transformer whose representation is computed
with Shifted windows. Swin Transformer can not only process high-resolution
images like CNN, but also has the advantage of Transformer to model long-range
dependency. In [24], an image restoration Transformer was developed based on
Swin Transformer, which achieves state-of-the-art performance in various image
reconstruction tasks.

##  3 Proposed Method

###  3.1 Model Formulation

In this paper, we use lowercase letters to denote scalars, use bold letters to
denote matrices, and use curlycue to denote tensors. Specially, let
$\mathcal{Y}\in\mathbb{R}^{W\times H\times s}$ represent the HR-MSI and
$\mathcal{Z}\in\mathbb{R}^{w\times h\times S}$ represent the LR-HSI. The goal
of HSISR is to combine information coming from an LR-HSI and an HR-MSI. The
former has high spectral resolution, with $S$ spectral bands, but low
geometric resolution with $w$ and $h$ being image width and height,
respectively. The latter has high spatial resolution with $W$ and $H$ being
image width and height, respectively, but a low spectral resolution $s$. We
aim to estimate a fused image $\mathcal{X}\in\mathbb{R}^{W\times H\times S}$
with both high spatial and high spectral resolution.

If we reshape the spatio-spectral data-cube $\mathcal{Y}$, $\mathcal{Z}$ and
$\mathcal{X}$ to their matrix formulations, _i.e._ ,
$\boldsymbol{Y}\in\mathbb{R}^{WH\times s}$,
$\boldsymbol{Z}\in\mathbb{R}^{wh\times S}$, and
$\boldsymbol{X}\in\mathbb{R}^{WH\times S}$, the observation models for the HR-
MSI and LR-HSI can be written as follows:

$\displaystyle\boldsymbol{Y}$ $\displaystyle=\boldsymbol{X}\boldsymbol{R},$
(1) $\displaystyle\boldsymbol{Z}$
$\displaystyle=\boldsymbol{C}\boldsymbol{X}.$ (2)

where $\boldsymbol{R}\in\mathbb{R}^{S\times s}$ is the spectral response
function of the multispectral sensor, $\boldsymbol{C}\in\mathbb{R}^{wh\times
WH}$ represents the degradation operator which is often assumed to be composed
of a cyclic convolution operator and a down-sampling matrix. We can obtain
$\boldsymbol{X}$ by solving the following optimization problem

$\displaystyle\mathop{\min}_{\boldsymbol{X}}\ \
\frac{1}{2}\lVert\boldsymbol{X}\boldsymbol{R}-\boldsymbol{Y}\rVert_{F}^{2}+\frac{1}{2}\lVert\boldsymbol{C}\boldsymbol{X}-\boldsymbol{Z}\rVert_{F}^{2}+\lambda
f(\boldsymbol{X}).$ (3)

where the first and second terms are fidelity terms and the third term is a
regularization term which represents prior knowledge of $\boldsymbol{X}$. In
the conventional HS/MS fusion methods, most of the priors are hand-crafted
based on empirical observation, such as sparse priors [21, 12], low-rank
priors [35, 51], self-similarity [19]. Although they have achieved promising
results, but they only use a certain property of hyperspectral images, and it
is difficult to apply to a variety of hyperspectral images. Pioneering works
[43, 37, 14] have proved that the performance of data-driven priors exceeds
that of hand-crafted priors. Therefore, in this work, we use a network to
learn the data-driven implicit prior.

Since the specific form of the prior is not specified, the regularization term
is usually non-smooth and non-differentiable functions of the outputs.
Therefore, in this paper we use a proximal gradient algorithm [4] to solve Eq.
(3). Let $\mathcal{L}(\boldsymbol{X})=g(\boldsymbol{X})+\lambda
f(\boldsymbol{X})$, where $g(\boldsymbol{X})$ is differentiable (includes the
first two terms of Eq. (3)) and $\lambda f(\boldsymbol{X})$ is non-
differentiable. The proximal gradient algorithm minimizes Eq. (3) by iterating
the following equation until convergence:

$\displaystyle\boldsymbol{X}^{(k+1)}$ $\displaystyle=\text{prox}_{\lambda\eta
f}(\boldsymbol{X}^{(k)}-\eta\bigtriangledown g(\boldsymbol{X}^{(k)})),$ (4)

where $\eta$ plays the role of step size, and $\text{prox}_{\lambda\eta
f}(\cdot)$ is a proximal operator dependent on $\lambda$, $\eta$ and $f$.
Since $\bigtriangledown
g(\boldsymbol{X}^{(k)})=(\boldsymbol{X}^{(k)}\boldsymbol{R}-\boldsymbol{Y})\boldsymbol{R}^{T}+\boldsymbol{C}^{T}(\boldsymbol{C}\boldsymbol{X}-\boldsymbol{Z})$,
we substitute it into Eq. (4) to obtain the following optimization problem
that can be solved iteratively:

$\displaystyle\boldsymbol{X}^{(k+1)}$ $\displaystyle=\text{prox}_{\lambda\eta
f}(\boldsymbol{X}^{(k)}-\eta(\boldsymbol{X}^{(k)}\boldsymbol{R}-\boldsymbol{Y})\boldsymbol{R}^{T}+\boldsymbol{C}^{T}(\boldsymbol{C}\boldsymbol{X}^{(k)}-\boldsymbol{Z})).$
(5)

###  3.2 Optimization-based Unfolding Network

Conventional proximal gradient algorithm requires many iterations to converge
and is computationally expensive. Moreover, the step size $\eta$ has to be
selected by hand and it is difficult to find the optimal $\eta$. To overcome
these shortcomings, recent works [13, 43, 14] used a deep unfolding network to
unfold the iterative optimization steps into a series of networks. These
models have been trend-setting and promising in solving the inverse problems.
The methods based on deep unfolding strategy have the fundamental advantages
of interpretability, fewer parameters, and fast convergence. We derive our
motivation from these works and design a deep unfolding network to solve the
optimization Eq. (5).

Before solving Eq. (5), we first present the physical meaning of matrix
operations in this paper: the left multiplication matrix represents the
spatial transformation of the input, and the right multiplication matrix
represents the channel transformation of the input. Then we unfold the
iteration Eq. (5) into the following equivalent four sequential parts:

$\displaystyle\boldsymbol{Y}^{(k)}=\boldsymbol{X}^{(k)}\boldsymbol{R}-\boldsymbol{Y},$
(6)
$\displaystyle\boldsymbol{Z}^{(k)}=\boldsymbol{C}\boldsymbol{X}^{(k)}-\boldsymbol{Z},$
(7)
$\displaystyle\hat{\boldsymbol{X}}^{(k)}=\eta(\boldsymbol{Y}^{(k)}\boldsymbol{R}^{T}+\boldsymbol{C}^{T}\boldsymbol{Z}^{(k)}),$
(8) $\displaystyle\boldsymbol{X}^{(k+1)}=\text{prox}_{\lambda\eta
f}(\boldsymbol{X}^{(k)}-\hat{\boldsymbol{X}}^{(k)}),$ (9)

where in Eq. (6), $\boldsymbol{R}\in\mathbb{R}^{S\times s}$ represents channel
decrease operator, which can be performed by using some convolution layers.
The simplest way is to apply the $1\times 1$ convolution to decrease the
channel number from $S$ to $s$. In this paper, we leverage the more flexible
$3\times 3$ convolution layer to achieve this. In Eq. (7),
$\boldsymbol{C}\in\mathbb{R}^{wh\times WH}$ represents the spatial
downsampling operator, we perform it by several $2\text{D}$ convolution layers
with kernel stride 2 and thus each $2\text{D}$ convolution layer carrying out
$2$ times downsampling. $\boldsymbol{R}^{T}\in\mathbb{R}^{s\times S}$ in Eq.
(8) represents channel increase operator, we perform it by using the $3\times
3$ convolution layers in a similar vein.
$\boldsymbol{C}^{T}\in\mathbb{R}^{HW\times wh}$ represents an upsampling
operator, and we perform it by several $2\text{D}$ transposed convolution
layers [16] with each $2\text{D}$ transposed convolution layer carrying out
$2$ times upsampling. Since it is almost impossible to find the best $\eta$ by
hand, we set $\eta$ to be a learnable parameter and update it as the backward
propagation of errors. In Eq. (9), $\text{prox}_{\lambda\eta f}(\cdot)$ is a
proximal operator used to model the image prior. For the simple prior
modeling, such as the $l$-norm regularization for sparse representation, the
proximal is tractable and can be modeled in closed-form through soft-
thresholding and shrinkage based unrolling network [50]. More details about
the prior network will be introduced in the next section.

Motivated by the proximal gradient iterations in Eq. (6)-(9), we can now
perform these invert linear inverse tasks by efficient network architectures.
Fig. 1 shows the architecture of the proposed network. The observations $Y$
and $Z$ forms the inputs for all iterations, which together with the output
$X^{(k)}$ of the previous iteration form the variable for the current
iteration. The network contains $K$ iteration stages, and each iteration stage
is composed of a data module (corresponding to Eq. (6)-(8)) and a prior module
(corresponding to Eq. (6)-(9)), which is modeled via an efficient network
architecture, as will be elaborated in the next section. In the first stage
(_i.e._ , $k=1$), we perform Bicubic upsampling on LR-HSI $Z$ to initialize
$\boldsymbol{X}^{(0)}$. To reduce the number of parameters, we share the
parameters for different stages. Therefore, the proximal map will be modeled
by unrolling the proximal gradient iterations through a recurrent-like
architecture [26]. To avoid information loss during transmissions between
iterations, inspired by some previous works [14, 42] we propose to densely
connect the hierarchical feature maps from the previous prior module to the
subsequent prior modules, as shown in Fig. 1.

Figure 2: Illustration of the network architecture of the proposed 3D-CNN and
Transformer Prior (3DT) module. LN and MLP, are the LayerNorm and MultiLayer
Perceptron layers, respectively. W-MSA and SW-MSA represent the window based
multi-head self-attention using regular and shifted window partitioning
configurations, respectively.

###  3.3 3D-CNN and Transformer Prior Network

For these deep unfolding networks, the proximal operator is usually modeled as
a deep neural model, which resembles the projection from the (noisy)
observation onto the manifold (where the true image lies in) of visually
plausible images. Therefore, the proximal operator can be seen as a denoiser
that gradually removes the artifacts from the noisy observation. Pioneering
works (_e.g._ , DBIN [37], MHF-net [43], MoG-DCN [14] _etc_.) usually used a
sub-network to learn this operation. In DBIN [37] and MHF-net [43], the
authors adopted ResNet to learn this proximal operator and attained
satisfactory performance. Recently, MoG-DCN [14] proposed to use U-net [30] as
the backbone network architecture to learn this proximal operator which
achieves the SOTA performance on the HS/MS fusion task. From these works, it
is easy to analyze that the performance of the selected prior estimation
network is the essential factor that determines the performance of the entire
HS/MS fusion network. Therefore, in this paper, we take the existing method
one step further and propose a new denoiser prior learning network based on
3D-CNN and Transformer network, which introduces the powerful Transformer [34]
structure into the prior learning network.

The architecture of the 3D-CNN and Transformer prior network is illustrated in
Fig. 2. The main components of the 3D-CNN and Transformer prior network are
Swin Transformer layer (STL) and 3D convolution layer. The STL was proposed
for the first time in [25]. It has long-range modeling capabilities like the
original Transformer, and the amount of calculations is much smaller than that
of the original Transformer. Therefore, it is a very good choice for learning
data priors. However, STL is designed for natural RGB images. It fully
considers the spatial self-similarity of the image, but does not consider the
spectral correlation of the HSI data. Therefore, it is not appropriate to
directly use STL to learn the priors of 3D HSIs. To address this problem, we
use 3D convolutional layers to learn the spectral priors of HSIs. Compared
with filters in 2D convolution, filters in 3D convolution are more flexible in
the channel dimension, making it more suitable for exploring the spatio-
spectral correlation of HSIs.

###  3.4 Training Details

We resort to the $L_{1}$ loss to train our network

$\displaystyle L$
$\displaystyle=\lVert\boldsymbol{X}^{(K)}-\boldsymbol{X}\rVert_{1},$ (10)

where $\boldsymbol{X}^{(K)}$ is the final output HR-HSI. $\boldsymbol{X}$ is
the ground truth HR-HSI.

In our implementation, we initialize $\eta$ with a uniform distribution in
$[0,1]$ and in each stage are updated with gradient-based method. We use Adam
optimizer [22] with $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$ to
train the network for 250000 iterations with a batch size of 8. We set each
epoch to have 2500 iterations, thus there are 100 epochs. The learning rate is
initialized as 0.0001. We implement and train our network using Pytorch
framework with an NVIDIA Tesla V100 GPU.

##  4 Experiments and Analyses

In this section, we compare our proposed method with eight other state-of-the-
art methods including four traditional methods, _e.g._ , Hysure
[31]111https://github.com/alfaiate/HySure, NSSR
[12]222http://see.xidian.edu.cn/faculty/wsdong/HSI_SR_Project.htm, CSTF
[23]333https://github.com/renweidian/CSTF, LTTR
[9]444https://github.com/renweidian/LTTR, and four recently proposed deep
learning-based methods, _e.g._ , CNN-FUS
[11]555https://github.com/renweidian/CNN-FUS, MHF-net
[43]666https://github.com/XieQi2015/MHF-net, DBIN
[37]777https://github.com/wwhappylife/Deep-Blind-Hyperspectral-Image-Fusion,
MoG-DCN [14]888https://github.com/chengerr/Model-Guided-Deep-Hyperspectral-
Image-Super-resolution. As for objective comparisons, we use four picture
quality indices (PQIs) to evaluate the performance of different methods,
including peak signal-to-noise ratio (PSNR), SAM [48], erreur relative globale
adimensionnelle de synthèse (ERGAS) [36] and structure similarity (SSIM) [39].
PSNR and SSIM are calculated on each 2D spatial image, which evaluate the
similarity between the recovered HSI and the ground truth based on MSE and
structural consistency, respectively. SAM calculates the average angle between
spectrum vectors of the recovered HSI and the ground truth. ERGAS reflects the
overall quality of the recovered HSI. Smaller values of ERGAS and SAM suggest
better results, while larger values of PSNR and SSIM imply better results.

Table 1: Influence of the number of iterations for our 3DT-Net. Model | PSNR | SAM | ERGAS | SSIM
---|---|---|---|---
3DT-Net ($K=1$) | 49.35 | 2.40 | 0.53 | 0.995
3DT-Net ($K=2$) | 50.26 | 2.13 | 0.50 | 0.996
3DT-Net ($K=3$) | 50.93 | 2.01 | 0.45 | 0.996
3DT-Net ($K=4$) | 50.91 | 2.05 | 0.46 | 0.996

###  4.1 Training Datasets

We evaluate our proposed method on two publicly simulated hyperspectral
imaging datasets including the _CAVE_
[45]999http://www.cs.columbia.edu/CAVE/databases/ and the _Harvard_
[3]101010http://vision.seas.harvard.edu/hyperspec/d2x5g3/. For real data
experiments, we use _WV-2_ 111111https://www.harrisgeospatial.com/Data-
Imagery/Satellite-Imagery/High-Resolution/WorldView-2.

The CAVE database consists of 32 indoor images captured under controlled
illumination with spatial size of $512\times 512$, including 31 spectral bands
of 10 nm wide, covering the visible spectrum from 400 to 700nm. We follow the
same setting as [14], the HR-MSI (RGB image) is generated by integrating all
the ground truth HR-HSI bands with the same simulated spectral response
function $R$ of a Nikon D700
camera121212http://www.maxmax.com/spectral_response.htm, and the LR-HSI can
obtained by first applying an anti-aliasing 8$\times$8 Gaussian filter with a
standard deviation of 2 to the original HR-HSIs followed by a downsampling
along both the horizontal and vertical dimensions with the scaling factor of
8. We select the first 20 samples and randomly extract $64\times 64$
overlapped patches from them as reference HR-HSIs for training. Thus, the HR-
HSIs, HR-MSIs and LR-HSIs are of sizes $64\times 64\times 31$, $64\times
64\times 3$, $8\times 8\times 31$, respectively. The remaining 12 samples of
the database are used for testing.

The Harvard database contains 50 indoor and outdoor images recorded under
daylight illumination with spatial size of $1,040\times 1,392$, including 31
spectral bands of 10 nm wide, covering the visible spectrum from 400 to 700nm.
Then we use the same method as CAVE database to get the HR-MSI and LR-HSI. We
select the first 30 samples and randomly extract $64\times 64$ overlapped
patches from them as reference HR-HSIs for training. Thus, the HR-HSIs, HR-
MSIs and LR-HSIs are of sizes $64\times 64\times 31$, $64\times 64\times 3$,
$8\times 8\times 31$, respectively. The remaining 20 samples of the database
are used for testing.

The WV-2 database contains a LR-HSI of size $419\times 658\times 8$ and an HR-
MSI of size $1676\times 2632\times 3$, while the ground truth HR-HSI is
unavailable. We follow the same setting as [43] to generate the training data.
We select the top half part of the HR-MSI $836\times 2632\times 3$ and LR-HSI
$209\times 658\times 8$ as training data and exploit the remaining parts of
the data set as testing data. We use the Wald’s protocol [49] to generate the
training samples. We divide the training data into $256\times 256\times 3$
overlapped HR-MSI patches and $64\times 64\times 8$ overlapped LR-HSI patches.
Then we down-sampling HR-MSI and LR-HSI patches into size $64\times 64\times
3$ and $16\times 16\times 8$ as the input HR-MSI and LR-HSI, respectively.
Note that the original $64\times 64\times 8$ HSI is taken as ground truth.

###  4.2 Ablation Study

In this section, we conduct simulated experiments to show the effectiveness of
our network in three aspects: the influence of stage number $K$, the
effectiveness of 3D-CNN and Transformer prior network, and the effectiveness
of 3D-CNN over 2D-CNN.

Table 2: Ablation study on the prior network with different setting. Network | Para(M) | PSNR | SAM | ERGAS | SSIM
---|---|---|---|---|---
ResNet | 6.75 | 48.29 | 2.51 | 0.60 | 0.995
U-net | 6.70 | 49.71 | 2.28 | 0.50 | 0.996
Transformer | 3.46 | 50.76 | 2.07 | 0.46 | 0.996
3DT-Net | 3.46 | 50.93 | 2.01 | 0.45 | 0.996
Figure 3: Qualitative results of the CAVE datasets at band 31. Top row: Ground
truth and reconstructed images by 6 comparison methods, with a demarcated
areas zoomed in 4 times for easy observation. Bottom row: RGB and the error
images of the result obtain by the 6 competing methods.

Effect of stage number $K$. To explore the impact of the number of unfolded
stages on the HSISR performance, we report the performance of the proposed
3DT-Net with different stage number $K$. Table 1 shows the average results
over 12 testing HSIs. Here, the bold values represent the best result, and the
results with underlines denote the second best. From the table, we can observe
that as the number of iterations increases, the performance of PSNR and SSIM
will increase, but when $K\textgreater 3$, the performance of PSNR and SSIM
will decrease. This shows that our method can quickly converge after several
iterations, but the performance may not be improved when the iteration is
increased, because the network is too complex and it will be difficult to
train to obtain the optimal solution. In the following experiments, we choose
$K=3$ in our implementation.

Table 3: Quantitative comparisons of different approaches over 10 testing images from the CAVE dataset with respect to four PQIs. Method | PSNR | SAM | ERGAS | SSIM
---|---|---|---|---
Hysure [31] | 40.06 | 9.66 | 1.30 | 0.976
NSSR [12] | 44.07 | 4.35 | 0.82 | 0.987
CSTF [23] | 42.41 | 5.04 | 0.87 | 0.979
LTTR [9] | 45.89 | 2.97 | 0.66 | 0.994
CNN-FUS [11] | 44.21 | 4.04 | 0.82 | 0.989
MHF-net [43] | 46.31 | 3.39 | 0.64 | 0.994
DBIN [37] | 48.82 | 2.09 | 0.50 | 0.996
MoG-DCN [14] | 49.89 | 2.04 | 0.45 | 0.996
3DT-Net | 50.93 | 2.01 | 0.45 | 0.996

Effect of 3D-CNN and Transformer prior network. To verify the effectiveness of
3D-CNN and Transformer prior network, we use the ResNet prior network adopted
in [43, 37] and U-net prior network adopted in [14] to replace the proposed
3D-CNN and Transformer prior network, respectively. Considering the powerful
long-range modeling ability of Transformer, it only needs very few parameters
to achieve strong performance. We use the U-net prior network proposed in [14]
as the best U-net prior network (its parameter amount may lager than our
method), rather than requiring that the parameters of the U-net prior network
must be the same as these of the 3D-CNN and Transformer prior network. For
fair comparison of ResNet, we let the parameters of ResNet prior network the
same as the U-net prior network. From Table 2, we can see that the PSNR value
of Transformer prior network is 1.05 dB higher than that of the U-net prior
network, and the parameters are only half of the U-net prior network. After
the introduction of 3D convolutional layers (the only difference between
Transformer and 3DT-Net), the PSNR performance increased again by 0.17 dB,
which proves that 3D convolutional layers can better extract the spatio-
spectral correlation priors.

Figure 4: Qualitative results of the Harvard datasets at band 31. Top row: Ground truth and reconstructed images by 6 comparison methods. Bottom row: RGB and the error images of the result obtain by the 6 competing methods. Table 4: Quantitative comparisons of different approaches over 20 testing images from the Harvard dataset with respect to four PQIs. Method | PSNR | SAM | ERGAS | SSIM
---|---|---|---|---
Hysure [31] | 44.26 | 3.75 | 1.40 | 0.983
NSSR [12] | 46.08 | 3.40 | 1.20 | 0.985
CSTF [23] | 44.98 | 3.54 | 1.07 | 0.980
LTTR [9] | 46.86 | 2.90 | 1.11 | 0.987
CNN-FUS [11] | 46.05 | 3.24 | 1.12 | 0.985
MHF-net [43] | 46.42 | 3.01 | 1.09 | 0.987
DBIN [37] | 47.36 | 2.71 | 0.97 | 0.988
MoG-DCN [14] | 47.64 | 2.67 | 0.91 | 0.988
3DT-Net | 48.05 | 2.27 | 0.84 | 0.991
Figure 5: Qualitative results of the WV-2 dataset. (a) and (b) are the HR-MSI
and LR-HSI of the right bottom area of Roman Colosseum. We show the composite
image of the HSI with bands 5-3-2 as R-G-B. (c)-(f) reconstructed images by 6
comparison methods, with a demarcated areas zoomed for easy observation.

###  4.3 Experiments with Simulated Data

The average results in terms of the PSNR, SAM, ERGAS and SSIM of all the
competing methods on the CAVE and Harvard datasets are reported in Table 3 and
Table 4, respectively. From these two tables, it can be seen that the
performance of deep learning-based methods are better than that of traditional
shallow learning-based methods, which shows the advantages and representation
ability of deep neural networks. Clearly, the proposed 3DT-Net method
outperforms all other competing methods by a considerable margin. The average
PSNR value of our method is more than 1.0 dB and 0.4 dB higher than that of
the second best method (_i.e._ , the MoG-DCN method) on CAVE and Harvard
datasets, respectively.

In Fig. 3 and Fig. 4, we show parts of the reconstructed HR-HSIs at 700 nm by
the competing methods for the test image real and fake apples from the CAVE
dataset and the test image imgf2 from the Harvard dataset, respectively. From
Fig. 3, we can see that the composite image obtained by 3DT-Net is the closest
to the ground-truth, while the results of Hysure and CNN-FUS contain obvious
incorrect structure or spectral distortion. From Fig. 4, we can see that all
the test methods can well reconstruct the HR spatial structures of the HSI.
Obviously, the proposed method performs best in recovering the details of the
original HSI and achieves the smallest reconstruction error.

###  4.4 Experiments with Real Data

To verify the robustness of our method on real data, a public dataset of real
MSIs called WV-2 are used in our experiments. In Fig. 5, we show a portion of
the fusion result of the testing data. It can be seen that MHF-net [43] and
DBIN [37] have obvious blur and artifacts. Visual inspection clearly shows
that the proposed method has better visual effect.

##  5 Conclusions

In this paper, we have provided a new deep unfolding network for hyperspectral
image super-resolution based on proximal gradient algorithm and Transformer
prior. Unlike other deep learning-based HSI super-resolution methods using CNN
to learn the prior of HSIs, our 3DT-Net use Transformer layers to exploit
spatial priors and 3D convolutional layers to exploit spatio-spectral
correlation priors. Compared with CNN, Transformer has long-range modeling
capabilities and can use fewer parameters to achieve better performance. 3D
convolution slides along the spatial and channel dimensions at the same time,
making it more suitable for exploring the spatio-spectral correlation of HSIs.
Evaluations on two public simulated datasets and one real datasets both
demonstrate that the proposed 3DT-Net achieves state-of-the-art performance in
terms of quantitative result and visual quality.

The proposed prior modeling method is general for many low-level image
restoration tasks. In particular, it can be seamlessly integrated with any
method under the Plug-and-Play framework, especially suitable for processing
high-dimensional data, such as hyperspectral image, video, and light-field.
The limitation of our approach is that it is a little more computationally
taxing than those convolutional residual block based network. Therefore, in
the future we will exploit some efficient implementations to extend our method
to other tasks.

## References

  * [1] Hamed Akbari, Luma Halig, David M Schuster, Baowei Fei, Adeboye Osunkoya, Viraj Master, Peter Nieh, and Georgia Chen.  Hyperspectral imaging and quantitative analysis for prostate cancer detection.  Journal of Biomedical Optics, 17(7):076005, 2012.
  * [2] Naveed Akhtar, Faisal Shafait, and Ajmal Mian.  Bayesian sparse representation for hyperspectral image super resolution.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3631–3640, 2015.
  * [3] Boaz Arad, Radu Timofte, Ohad Ben-Shahar, Yi-Tun Lin, and Graham D Finlayson.  Ntire 2020 challenge on spectral reconstruction from an rgb image.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 446–447, 2020.
  * [4] Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  SIAM journal on imaging sciences, 2(1):183–202, 2009.
  * [5] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le.  Attention augmented convolutional networks.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 3286–3295, 2019.
  * [6] Gustavo Camps-Valls, Devis Tuia, Lorenzo Bruzzone, and Jon Atli Benediktsson.  Advances in hyperspectral image classification: Earth monitoring with statistical learning methods.  IEEE Signal Processing Magazine, 31(1):45–54, 2013.
  * [7] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12299–12310, 2021.
  * [8] Renwei Dian, Leyuan Fang, and Shutao Li.  Hyperspectral image super-resolution via non-local sparse tensor factorization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5344–5353, 2017.
  * [9] Renwei Dian, Shutao Li, and Leyuan Fang.  Learning a low tensor-train rank representation for hyperspectral image super-resolution.  IEEE transactions on neural networks and learning systems, 30(9):2672–2683, 2019.
  * [10] Renwei Dian, Shutao Li, Anjing Guo, and Leyuan Fang.  Deep hyperspectral image sharpening.  IEEE Transactions on Neural Networks and Learning Systems, (99):1–11, 2018.
  * [11] Renwei Dian, Shutao Li, and Xudong Kang.  Regularizing hyperspectral and multispectral image fusion by cnn denoiser.  IEEE transactions on neural networks and learning systems, 32(3):1124–1135, 2020.
  * [12] Weisheng Dong, Fazuo Fu, Guangming Shi, Xun Cao, Jinjian Wu, Guangyu Li, and Xin Li.  Hyperspectral image super-resolution via non-negative structured sparse representation.  IEEE Transactions on Image Processing, 25(5):2337–2352, 2016.
  * [13] Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fangfang Wu, and Xiaotong Lu.  Denoising prior driven deep neural network for image restoration.  IEEE transactions on pattern analysis and machine intelligence, 41(10):2305–2318, 2018.
  * [14] Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, and Xin Li.  Model-guided deep hyperspectral image super-resolution.  IEEE Transactions on Image Processing, 2021.
  * [15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [16] Vincent Dumoulin and Francesco Visin.  A guide to convolution arithmetic for deep learning.  arXiv preprint arXiv:1603.07285, 2016.
  * [17] Mathieu Fauvel, Yuliya Tarabalka, Jon Atli Benediktsson, Jocelyn Chanussot, and James C Tilton.  Advances in spectral-spatial classification of hyperspectral images.  Proceedings of the IEEE, 101(3):652–675, 2012.
  * [18] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014.
  * [19] Xian-Hua Han, Boxin Shi, and Yinqiang Zheng.  Self-similarity constrained sparse representation for hyperspectral image super-resolution.  IEEE Transactions on Image Processing, 27(11):5625–5637, 2018.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [21] Rei Kawakami, Yasuyuki Matsushita, John Wright, Moshe Ben-Ezra, Yu-Wing Tai, and Katsushi Ikeuchi.  High-resolution hyperspectral imaging via matrix factorization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2329–2336, 2011.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Shutao Li, Renwei Dian, Leyuan Fang, and José M Bioucas-Dias.  Fusing hyperspectral and multispectral images via coupled sparse tensor factorization.  IEEE Transactions on Image Processing, 27(8):4118–4130, 2018.
  * [24] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1833–1844, 2021.
  * [25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  arXiv preprint arXiv:2103.14030, 2021.
  * [26] Morteza Mardani, Qingyun Sun, Shreyas Vasawanala, Vardan Papyan, Hatef Monajemi, John Pauly, and David Donoho.  Neural proximal gradient descent for compressive imaging.  arXiv preprint arXiv:1806.03963, 2018.
  * [27] Frosti Palsson, Johannes R Sveinsson, and Magnus O Ulfarsson.  A new pansharpening algorithm based on total variation.  IEEE Geoscience and Remote Sensing Letters, 11(1):318–322, 2013\.
  * [28] Zhihong Pan, Glenn Healey, Manish Prasad, and Bruce Tromberg.  Face recognition in hyperspectral images.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12):1552–1560, 2003.
  * [29] Adnan Qayyum, Syed Muhammad Anwar, Muhammad Awais, and Muhammad Majid.  Medical image retrieval using deep convolutional neural network.  Neurocomputing, 266:8–20, 2017.
  * [30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [31] Miguel Simoes, José Bioucas-Dias, Luis B Almeida, and Jocelyn Chanussot.  A convex formulation for hyperspectral image superresolution via subspace-based regularization.  IEEE Transactions on Geoscience and Remote Sensing, 53(6):3373–3388, 2014.
  * [32] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani.  Bottleneck transformers for visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16519–16529, 2021.
  * [33] Hien Van Nguyen, Amit Banerjee, and Rama Chellappa.  Tracking via object reflectance using a hyperspectral video camera.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 44–51, 2010.
  * [34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [35] Miguel A Veganzones, Miguel Simoes, Giorgio Licciardi, Naoto Yokoya, José M Bioucas-Dias, and Jocelyn Chanussot.  Hyperspectral super-resolution of locally low rank images from complementary multisource data.  IEEE Transactions on Image Processing, 25(1):274–288, 2015.
  * [36] Lucien Wald.  Data fusion: definitions and architectures: fusion of images of different spatial resolutions.  Presses des MINES, 2002.
  * [37] Wu Wang, Weihong Zeng, Yue Huang, Xinghao Ding, and John Paisley.  Deep blind hyperspectral image fusion.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4150–4159, 2019.
  * [38] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803, 2018.
  * [39] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [40] Qi Wei, José Bioucas-Dias, Nicolas Dobigeon, and Jean-Yves Tourneret.  Hyperspectral and multispectral image fusion based on a sparse representation.  IEEE Transactions on Geoscience and Remote Sensing, 53(7):3658–3668, 2015.
  * [41] Qi Wei, José Bioucas-Dias, Nicolas Dobigeon, Jean-Yves Tourneret, Marcus Chen, and Simon Godsill.  Multiband image fusion based on spectral unmixing.  IEEE Transactions on Geoscience and Remote Sensing, 54(12):7236–7249, 2016.
  * [42] Zhuoyuan Wu, Jian Zhang, and Chong Mou.  Dense deep unfolding network with 3d-cnn prior for snapshot compressive imaging.  arXiv preprint arXiv:2109.06548, 2021.
  * [43] Qi Xie, Minghao Zhou, Qian Zhao, Zongben Xu, and Deyu Meng.  Mhf-net: An interpretable deep network for multispectral and hyperspectral image fusion.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [44] Yang Xu, Zebin Wu, Jocelyn Chanussot, and Zhihui Wei.  Nonlocal patch tensor sparse representation for hyperspectral image super-resolution.  IEEE Transactions on Image Processing, 28(6):3034–3047, 2019.
  * [45] Fumihito Yasuma, Tomoo Mitsunaga, Daisuke Iso, and Shree K Nayar.  Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum.  IEEE Transactions on Image Processing, 19(9):2241–2253, 2010.
  * [46] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu.  Disentangled non-local neural networks.  In European Conference on Computer Vision, pages 191–207. Springer, 2020.
  * [47] Naoto Yokoya, Takehisa Yairi, and Akira Iwasaki.  Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion.  IEEE Transactions on Geoscience and Remote Sensing, 50(2):528–537, 2011.
  * [48] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman.  Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm.  1992\.
  * [49] Yongnian Zeng, Wei Huang, Maoguo Liu, Honghui Zhang, and Bin Zou.  Fusion of satellite images in urban area: Assessing the quality of resulting images.  In 2010 18th International Conference on Geoinformatics, pages 1–4. IEEE, 2010.
  * [50] Jian Zhang and Bernard Ghanem.  ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1828–1837, 2018.
  * [51] Kai Zhang, Min Wang, and Shuyuan Yang.  Multispectral and hyperspectral image fusion based on group spectral embedding and low-rank factorization.  IEEE Transactions on Geoscience and Remote Sensing, 55(3):1363–1371, 2016.
  * [52] Lei Zhang, Wei Wei, Chengcheng Bai, Yifan Gao, and Yanning Zhang.  Exploiting clustering manifold structure for hyperspectral imagery super-resolution.  IEEE Transactions on Image Processing, 27(12):5969–5982, 2018.