# Full-resolution quality assessment for pansharpening

Giuseppe Scarpa  , Matteo Ciotola
Department of Electrical Engineering and Information Technology (DIETI)
University Federico II
80125 Naples, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

A reliable quality assessment procedure for pansharpening methods is of
critical importance for the development of the related solutions.
Unfortunately, the lack of ground-truths to be used as guidance for an
objective evaluation has pushed the community to resort to two approaches
which can also be jointly applied. Hence, two kinds of indexes can be found in
the literature: i) reference-based reduced-resolution indexes aimed to assess
the synthesis ability; ii) no-reference subjective quality indexes for full-
resolution datasets aimed to assess spectral and spatial consistency. Both
reference-based and no-reference indexes present critical shortcomings which
motivate the community to explore new solutions. In this work, we propose an
alternative no-reference full-resolution assessment framework. On one side we
introduce a protocol, namely the reprojection protocol, to take care of the
spectral consistency issue. On the other side, a new index of the spatial
consistency between the pansharpened image and the panchromatic band at full
resolution is also proposed. Experimental results carried out on different
datasets/sensors demonstrate the effectiveness of the proposed approach.

_Keywords_ Super-resolution; image enhancement; convolutional neural networks;
data fusion; multispectral images.

##  1 Introduction

Image pansharpening is the process of merging two observations of the same
scene, a low resolution multispectral (MS) component and a high resolution
panchromatic (PAN) component, to generate a new multispectral image that
displays both the rich spectral content of the MS and the high resolution of
the PAN. By following the taxonomy proposed in [79], pansharpening methods can
be roughly grouped in four main categories, component substitution (CS) [70],
multiresolution analysis (MRA) [60], variational optimization (VO) [82, 57],
and machine/deep learning (ML) [45, 91].

In the CS approach, the multispectral image is transformed in a suitable
domain where one of its components is replaced with the PAN. In the particular
case of three spectral bands, the Intensity-Hue-Saturation (IHS) transform is
an option where the intensity component can be replaced with the PAN band
[74]. This method has been generalized in [73] (GIHS) to handle a larger
number of bands. Other useful transforms to implement a CS solution include
the principal component analysis [15], the Brovey transform [27] and the Gram-
Schmidt (GS) decomposition [37]. More recently, adaptive CS methods have also
been proposed, such as the advanced versions of GIHS and GS [5], the partial
replacement CS method (PRACS) [17], or the band-dependent spatial detail
(BDSD) injection method and its variants [26, 24, 77].

In MRA approaches [60], the pansharpening task is addressed from the
perspective of a pyramidal decomposition to separate low-frequency content
from detail components. The high frequency spatial details are extracted by
means of a multi-resolution decomposition, such as decimated or undecimated
wavelet transforms [53, 60, 54, 32], Laplacian pyramids [2, 3, 4, 40, 63], or
other nonseparable transforms, e.g., contourlet [67]. Extracted details are
then properly injected into the upscaled MS component.

A further set of methods address the pansharpening problem through the
variational optimization of suitable models of the fused image. In [82] the
optimization target involves the degradation filters mapping high-resolution
to low-resolution images, while [75] leverage on sparse representations for
detail injection. Palsson et al. proposed several methods of this class. A
total variation regularized least square formulation is provided in [56]. The
same research team has framed the pansharpening as a maximum a posteriori
problem in [55] and, more recently, explored the use of low-rank
representations of the joint PAN-MS [57]. Other methods do not fit the above
categories and can be roughly classified as statistical [23, 94, 48, 69, 97],
dictionary-based [42, 41, 101, 16, 100, 29], or matrix factorization
approaches [92, 38, 30]. The reader is referred to [78, 79] for a more
comprehensive review.

In recent years, a paradigm shift from model-based to data-driven approaches
has revolutionized all fields of image processing, from computer vision [34,
20, 28, 39, 96] to remote sensing [91, 65, 13, 47]. In pansharpening, the
first method based on convolutional neural networks (CNN) was proposed by Masi
et al. in 2016 [45], after which many more followed in a few years’ span [91,
88, 89, 62, 46, 11, 93, 43, 68, 76, 95, 22, 21]. It seems safe to say that
deep learning is currently the most popular approach for pansharpening.
Nonetheless, it suffers from a major problem: the lack of ground truth data
for supervised training. In fact, multi-resolution sensors can only provide
the original PAN-MS data, downgraded in space or spectrum, never their high-
resolution versions, which remain to be estimated.

Based on this brief and certainly not exhaustive overview, it appears that
pansharpening is a very active research field, with many new methods proposed
every year. Reliable quality assessment procedures are of critical importance
for correctly advancing the state of the art, and a wrong evaluation paradigm
may negatively impact the design or tuning of any new solution. Unfortunately,
by the very nature of pansharpening, no Ground Truth (GT) data are available
to perform a reference-based assessment. As a consequence two kinds of quality
assessments are usually employed:

  * i)

reference-based reduced-resolution assessment (synthesis check)

  * ii)

no-reference full-resolution assessment (consistency check).

Lacking GTs, the synthesis capabilities of any pansharpening method can only
be assessed on “synthetic” data. In particular, Wald’s protocol [86] suggests
taking the real PAN-MS data and applying a proper resolution downgrade process
for scale reduction. The scaled PAN-MS pair is the synthesized image whose GT
is now given by the original MS component. The way the resolution downgrade
has to be performed has been object of intense research in the last two
decades and has a non negligible impact on the reliability of the consequent
quality assessment. However, no matter how a GT is obtained, there exist
plenty of reference-based image quality indicators. The spectral angle mapper
was introduced in [35] and assesses the balance among the spectral bands. On
the contrary, the spatial correlation coefficient [98] computes the
correlation coefficient across the high-pass filtered bands of the
pansharpened image and of the GT. It is therefore oriented to the assessment
of the spatial quality. The Erreur Relative Globale Adimensionnelle de
Synthése [85] generalizes the root mean squared error, introducing band-wise
correction weights. The universal image quality index [87] compares image
statistics to take into account local correlation, intensity and contrast.
Based on this general purpose image quality index, domain specific variants
suitably adapted to the pansharpening have been proposed in [7, 25]. In
addition to these indexes, other popular general purpose options are sometimes
considered for pansharpening, e.g., the peak signal-to-noise ratio or the
structural similarity (SSIM) index.

Besides, spectral or spatial consistency indexes suited for full-resolution
images which do not require any GT have also been developed. A first spectral
distortion index proposed by Zhou et al. [98] compares a rescaled version of
the pansharpened image with the MS image. Other spectral distortion indexes
follow a similar procedure [6, 51]. In [58] the Quality for Low Resolution
(QLR) index based on SSIM was proposed and later updated exchanging SSIM with
the composite image quality measure CMSC based on means, standard deviations
and correlation coefficient [59]. Basically, most of the known spectral
distortion indexes follow this protocol: degradation of the pansharpened image
and comparison with the MS image. The different behavior stems from the
degradation model and the error measure. For example [58] is based on the
structural similarity indexes while [59] leverages on a composite image
quality measure based on statistics such as mean, variance and correlation
coefficient. On the other hand, the spatial consistency can be checked through
the cross-scale invariance of some statistics [6]. A similar approach is
proposed in [51] where only high-frequency components are concerned. In [58]
it is proposed a Quality for High Resolution (QHR) index which assesses the
SSIM index between the panchromatic band and a projection of the pansharpened
image in the PAN domain. A variant of this approach with a different error
term was proposed in [59]. Another similar approach is proposed in [10], where
the coefficient of determination is used to compare the PAN image with its
projection from the pansharpened image. It is also worth to mention other no-
reference quality indexes, such as the Natural Image Quality Evaluator (NIQE)
[50, 36] that seeks to assess the image quality by-itself rather than checking
consistency.

Finally, often the spectral and the spatial consistency indexes are somehow
combined, e.g. through geometric means, to provide a single hybrid consistency
index representing the overall quality no-reference assessment of the fused
images [6, 51, 58, 59, 1, 36, 49, 14, 84, 83].

Both reference-based and no-reference indexes present inherent limitations
which are analyzed in the next Section.

In this work, we propose new full-resolution quality indexes that overcome
some of these problems and provide a more reliable guidance for the
development of ever more accurate pansharpening methods. To this purpose, the
developed indexes have been made available to the community through a web
repository at https://github.com/matciotola/fr-pansh-eval-tool/.

The remainder of the paper is organized as follows. Section 2 provides a brief
critical survey on pansharpening assessment. Section 3 introduces the proposed
approach. Section 4 discusses the experimental results and, finally, Section 5
draws conclusions.

##  2 A review of panshapening indexes

Due to the lack of ground truths, visual inspection by human experts is the
ultimate benchmark for pansharpening methods. However, this is a lengthy and
tedious work, and numerical indexes are essential for a viable development
process. Reduced-Resolution (RR) indexes are computed, according to Wald’s
protocol [86], by reducing the scale of both MS and PAN components. Then,
pansharpening is performed on these rescaled data, and the original MS is used
as GT to compute reference-based indexes. Instead, Full-Resolution (FR)
indexes are computed on the original data and try to assess their spectral
and/or spatial consistency with the pansharpened output. In a comprehensive
evaluation, following Wald’s protocol [86], usually, both types of indexes are
considered together in addition to visual inspection [79]. In order to
highlight the critical points of the current approaches to the quality
assessment, we provide a brief overview of some of the most representative
indexes in the following.

###  2.1 Reduced-resolution assessment

Let $M$ and $P$ be the original MS and PAN components, respectively, and
$\hat{M}$ be the pansharpened image. The synthesis properties can not be
directly verified on the full-resolution pair $(M,P)$ due to the lack of the
GT with which to compare $\hat{M}$. Leveraging on a scale invariance
hypothesis, it is however possible to synthesize datasets with GT by properly
downgrading the available $(M,P)$ pairs by a factor $R$ (PAN-MS resolution
ratio). The resulting reduced resolution pair
$(M_{\downarrow},P_{\downarrow})$ can therefore be used as the source input
for the pansharpening task, whereas the original MS image $M$ plays as GT.
Once moved to the reduced resolution space, many different quality indexes can
be computed to assess the mismatch between the pansharpened image and its
reference GT, such as root mean square error, structural similarity, and so
on. In particular, referring to some of the most popular ones for
pansharpening, the most common are the following.

  * a)

SAM (Spectral Angle Mapper) [35]. It determines the spectral similarity in
terms of pixel-wise average angle between spectral signatures. Said
$\mathbf{v}$ and $\mathbf{\hat{v}}$ two corresponding pixel spectral responses
to be compared, SAM is obtained by averaging over all image locations the
following “angle” among vectors:

${\rm
SAM}(\mathbf{v},\mathbf{\hat{v}})=\arccos\left(\frac{\left\langle\mathbf{v},\mathbf{\hat{v}}\right\rangle}{\|\mathbf{v}\|_{2}\cdot\|\mathbf{\hat{v}}\|_{2}}\right).$
(1)

  * b)

ERGAS (Erreur Relative Globale Adimensionnelle de Synthése) [85]. It is one of
the most popular indexes to assess both spectral and structural fidelity
between a synthesized image and a target GT. Such an index presents
interesting invariance properties. Indeed, it is insensitive to radiometric
range, number of bands and resolution ratio. Said $B$ the number of spectral
bands, it is defined as

${\rm ERGAS}=\frac{100}{R}\sqrt{\frac{1}{B}\sum_{b=1}^{B}\left(\frac{{\rm
RMSE}_{b}}{\mu^{\rm GT}_{b}}\right)^{2}},$ (2)

where ${\rm RMSE}_{b}$ is the root mean square error over the $b$-th spectral
band and $\mu^{\rm GT}_{b}$ is the average intensity of the $b$-th band of the
GT image.

  * c)

$Q2^{n}$ [25]. It is a multiband extension of the universal image quality
index (UIQI) [87]. Each pixel of an image with $B$ spectral bands is
accommodated into a hypercomplex (HC) number with one real part and $B$– 1
imaginary parts. Let $\mathbf{z}$ and $\mathbf{\hat{z}}$ denote the HC
representations of a generic GT pixel and its prediction, respectively, then
$Q2^{n}$ can be written as the product of three terms:

$Q2^{n}=\frac{|\sigma_{\mathbf{z}\mathbf{\hat{z}}}|}{\sigma_{\mathbf{z}}\sigma_{\mathbf{\hat{z}}}}\cdot\frac{2\sigma_{\mathbf{z}}\sigma_{\mathbf{\hat{z}}}}{\sigma_{\mathbf{z}}^{2}+\sigma_{\mathbf{\hat{z}}}^{2}}\cdot\frac{2\mu_{\mathbf{z}}\mu_{\mathbf{\hat{z}}}}{|\mu_{\mathbf{z}}|^{2}+|\mu_{\mathbf{\hat{z}}}|^{2}}.$
(3)

The first factor provides the modulus of the HC correlation coefficient
between $\mathbf{z}$ and $\mathbf{\hat{z}}$. The second and the third terms
measure contrast changes and mean bias, respectively, on all bands
simultaneously. Statistics are typically computed on 32$\times$32 pixels
blocks, then and an average over the blocks of the whole image provides the
global $Q2^{n}$ score, which takes values in the $[0,1]$ interval, being 1 the
optimal value achieved if and only if $\mathbf{z}=\mathbf{\hat{z}}$ in each
location.

Behind its appealing simplicity, regardless of the specific error measurement
employed, the reduced resolution assessment hides some major subtle pitfalls
that undermine its usefulness. On one hand, its accuracy depends critically on
the method used for scaling, which may not correspond to the actual sensing
conditions. More fundamentally, it relies on the arbitrary assumption that a
method optimized for the RR scale will keep its good behavior at the FR scale.
Experimental evidence shows this not to be the case. Furthermore, optimizing
parameters for a given scale may lead to a sort of “scale overfitting”, with
the perverse effect of degrading performance on a different scale. For these
reasons, many recent studies focus on no-reference full-resolution quality
indexes [36, 10, 84, 49]. It is also worth to recall that the assessment of
spectral and spatial consistencies is not necessarily linked to a fusion task
and is of a broader interest, with applications in the medical [71],
agricultural [33] and food [99] sciences, to mention a few.

###  2.2 Full-resolution no-reference assessment

As a consequence of the above limitations, many studies have moved toward a
no-reference approach to the pansharpening quality assessment. In particular,
the same Wald’s protocol [86] recognizes the problem invoking a complementary
consistency quality check in addition to the synthesis capacity assessment. In
practice, for consistency-based assessment, the input data are pansharpened in
their original (full) resolution space. The outcome is then compared with both
input components, MS and PAN, for spectral and consistency assessment,
respectively. The spectral consistency check requires a spatial resolution
degradation of the pansharpened image, which is typically done by means of a
Low-Pass Filtering (LPF) followed by decimation, using LPFs that approximate
the sensor Modulation Transfer Function (MTF) [9]. Other approaches have also
been explored, such as wavelet transforms [19, 61], averaging operators [52],
pyramid decomposition [2]. The spatial (or structural) consistency check is
indeed more controversial lacking a clear relationship that allows to project
the pansharpened MS in the PAN domain [72]. No-reference indexes are also
often referred to as FR indexes, as they do not require any resolution
degradation of the input data, contrarily to the reference-based ones which do
need it because of the lack of GTs. For this reason, the latter are also
referred to as RR indexes.

Among the several FR quality assessment approaches proposed in the last years
[6, 98, 51, 67, 10], we will recall a few of them which are the most
frequently used. In particular, Alparone et al. [6] proposed a spectral
distortion index $D_{\lambda}$ defined as

$D_{\lambda}=\sqrt[p]{\frac{1}{B(B-1)}\sum_{i=1}^{B}\sum_{j=1,j\neq
j}^{B}\left|d_{i,j}(M,\hat{M})\right|^{p}},$ (4)

where $d_{i,j}(M,\hat{M})=Q(M_{i},M_{j})-Q(\hat{M}_{i},\hat{M}_{j})$, with
subscripts indicating selected bands and being $Q(\cdot,\cdot)$ the UIQI [87].
Such an index was also enclosed in the pansharpening benchmark toolbox [78].
As it can be observed, $D_{\lambda}$ compares the inter-band relationship
through the UIQI, separately for the input MS $M$ and the pansharpened image
$\hat{M}$, and then quantifies their divergence over each coupling of spectral
bands. As it can be figured, lacking a direct measurement of the distance
between $M$ and $\hat{M}$, and leveraging on an assumption of cross-scale
inter-band correlation invariance only, there can be important spectral
aberrations that can remain unseen by $D_{\lambda}$. The recently published
revisited version [79] of the same toolbox [78], indeed, makes use of an
approximated version of Khan’s index proposed in [51] to monitor the spectral
consistency. In particular, Khan’s index is defined as follows

$D_{\lambda}^{(\rm K)}=1-Q2^{n}\left(\hat{M}_{\downarrow}^{\rm lp},M\right),$
(5)

where $\hat{M}_{\downarrow}^{\rm lp}$ indicates the MTF-based low-pass
filtered and decimated version of the pansharpened image $\hat{M}$, while $M$
is the original input MS. Its approximation used in the toolbox [79], name it
$\widetilde{D}_{\lambda}^{(\rm K)}$, avoids the decimation of $\hat{M}$ by
using an upscaled version $\widetilde{M}$ of $M$:

$\widetilde{D}_{\lambda}^{(\rm K)}=1-Q2^{n}\left(\hat{M}^{\rm
lp},\widetilde{M}\right),$ (6)

being $\hat{M}^{\rm lp}$ the low-pass version of $\hat{M}$. These indexes
range between 0 (optimal value) and 1, and clearly relate to the spectral
consistency, since the resolution downgrade process applied to $\hat{M}$ gets
rid of the high-pass content retrieved from the PAN. However, two critical
points should be taken into account:

  * a)

dependence on the accuracy of the estimated MTF;

  * b)

sensitivity to the PAN-MS alignment.

In particular, as a consequence of b), methods such as several CS approaches
[44, 37, 5, 63, 17] that intrinsically address the coregistration problem are
penalized by both $D_{\lambda}^{(\rm K)}$ and $\widetilde{D}_{\lambda}^{(\rm
K)}$ on misaligned datasets. Hence, in order to fairly use Khan’s index it is
usually recommended to keep separate the registration and pansharpening
problems by means of a prior coregistration of the datasets to be used for
testing [78, 79].

Several indexes have also been proposed for FR spatial consistency check [6,
51, 58, 10]. In particular, the spatial distortion index proposed in [6] is
used in the most recent pansharpening assessment toolbox [79] and is defined
as:

$D_{\rm
S}=\left\lVert\frac{1}{B}\sum_{i=1}^{B}Q(\hat{M}_{i},P)-Q(M_{i},P_{\downarrow})\right\rVert_{q},$
(7)

where $P_{\downarrow}$ is the original PAN component resized to the MS scale
and $Q(\cdot,\cdot)$ is the UIQI quality index [87]. This index is somehow
similar to $D_{\lambda}$ as it checks the cross-scale invariance of the PAN-MS
spectral correlation. Therefore it inherits similar limitations, which are:

  * i.

no direct comparison between the pansharpened image $\hat{M}$ and the PAN $P$;

  * ii.

a cross-scale invariance assumption for which there are no guarantees.

##  3 Proposed full-resolution indexes

We now propose some new indexes for assessing the quality of pansharpening,
aiming to overcome some limitations of traditional ones. Given the weakness of
the scale-invariance assumption of reduced-resolution quality assessment
indexes, we focus on full-resolution ones, moving our perspective from
synthesis to consistency. In the following, we will discuss spectral and
spatial assessment separately.

###  3.1 Reprojection protocol for spectral accuracy assessment

The assessment of the spectral consistency is not as controversial as the
assessment of the spatial one. In particular, as far as we know, Khan’s
approach is the most trusted one for this purpose. Nonetheless, as remarked
above, it has some limitations and, in particular, in order to work properly,
the datasets must be well aligned.

To cope with this issue without the need to operate any coregistration on the
available datasets, let us now consider a variation of Khan’s index where the
decimation of $\hat{M}^{\rm lp}$ is performed using a band-wise shift that
maximizes the correlation coefficient between the low-pass filtered PAN
$P^{\rm lp}$ and each ideally interpolated MS band $\widetilde{M}_{b}$. In
case of aligned datasets, it returns the usual Khan’s index because no shifts
are applied. Otherwise, before decimation, we apply to $\hat{M}$ the same
shift that would align $\widetilde{M}$ and $P^{\rm lp}$. Let us indicate as
$D_{\lambda,\rm align}^{(\rm K)}$ such an index which is formally given by

$D_{\lambda,\rm align}^{(\rm K)}=1-Q2^{n}\left(\hat{M}_{\downarrow a}^{\rm
lp},M\right),$ (8)

where ${\downarrow}{a}$ indicates decimation with band-wise alignment.

Let $\mathcal{M}$ indicate a generic reference-based distortion index, such as
$Q2^{n}$, SAM or ERGAS. We define the reprojection index R-$\mathcal{M}$ as

$\mbox{R-}\mathcal{M}=\mathcal{M}\left(\hat{M}^{\rm lp}_{\downarrow
a},M\right),$ (9)

that in the particular case of $\mathcal{M}=Q2^{n}$ reduces to the complement
of the $D_{\lambda,\rm align}^{(\rm K)}$ variant defined above (Eq. 8), i.e.,

$\mbox{R-}Q2^{n}=Q2^{n}\left(\hat{M}_{\downarrow a}^{\rm
lp},M\right)=1-D_{\lambda,\rm align}^{(\rm K)}.$ (10)

In other words, the pansharpened image $\hat{M}$ is projected in the original
MS space prior to be compared with the MS image $M$ (reference) using the
error index $\mathcal{M}$. The projection accounts for both eventual
misalignment and sensor MTF using matched low-pass filters. Of course, because
of the low-pass filtering, the high frequency image content (spatial details)
is discarded and some further indexes will be necessary to assess spatial
quality. Note also that, under the hypothesis that the sensor MTF is correctly
simulated, the reprojection indexes ensure that an ideal pansharpening (GT)
would achieve the best score as expected. This will be experimentally analyzed
in Sec. 4.3.

Figure 1: Spectral consistency check using reprojection (clockwise) vs
synthesis check according to Wald’s protocol (counterclockwise). $\mathcal{M}$
is any reference-based index. In gray and red are shown the PAN and MS
channels, respectively.

A pictorial representation of the proposed reprojection indexes which relates
them to the synthesis assessment protocol proposed by Wald [86] is given in
Fig.1. The common starting point (top-left) is the available $(P,M)$ pair.
Indexes based on Wald’s protocol follow the counterclockwise path: decimation-
pansharpening-measure. The proposed reprojection protocol, instead, consists
in following the clockwise path: pansharpening-decimation-measure, where the
actual pansharpened image is first computed, and then reprojected on the low-
resolution domain for quality assessment. Eventually, in both cases, the
original MS is used as a reference for any error measurement. In our proposal,
however, pansharpening precedes the resolution downgrading, thereby exploiting
the most valuable source of spatial information, the original PAN.
Furthermore, the overall evaluation no longer depends on how the PAN is
downscaled. Since the the reprojection protocol involves only the low-
resolution component of the synthesized image, it has to be considered a
spectral consistency rather than a synthesis check. A pseudo-code of the
reprojection indexes is given in Algorithm 1.

1:$\hat{M}$, $M$, $P$, $\mathbf{g}_{\rm MTF}$ (MTF parameters), $\mathcal{M}$
(error function)

2:R-$\mathcal{M}$

3:$\widetilde{M}={\rm upscale}(M,R)$ $\triangleright$ polynomial interpolation

4:$P^{\rm lp}={\rm LPF}(P,\mathbf{g}_{\rm MTF})$ $\triangleright$ MTF-tailored
low-pass filtering

5:for $b=1:B$ do

6:
$\mathbf{s}(b)=\underset{(m,n)\in[1,R]^{2}}{\arg\max}\mbox{corrcoef}\left(P^{\rm
lp}_{i,j},\widetilde{M}(b)_{i+m,j+n}\right)$ $\triangleright$ band-wise shift
for alignment

7:end for

8:$\hat{M}^{\rm lp}={\rm LPF}(\hat{M},\mathbf{g}_{\rm MTF})$

9:$\hat{M}_{\downarrow a}^{\rm lp}={\rm decimation}(\hat{M}^{\rm
lp},R,\mathbf{s})$

10:$\mbox{R-}\mathcal{M}=\mathcal{M}\left(\hat{M}^{\rm lp}_{\downarrow
a},M\right)$ $\triangleright$ Eq. 9

Algorithm 1 Reprojection error assessment

###  3.2 Correlation-based spatial consistency index

In order to complement the reprojection-based assessment, we also propose a
new index to evaluate the spatial consistency with the aim of obtaining
indications better correlated to human judgment than those provided by $D_{\rm
S}$. In particular, to assess the preservation of fine-scale spatial
structures, we compute the average local correlation between the pansharpened
image and the PAN.

Let $X^{\sigma}_{ij}$ indicate a $\sigma\times\sigma$ patch of $X$ centered on
location $(i,j)$. We compute the correlation field $\rho^{\sigma}_{P\hat{M}}$
given by the local correlation coefficients between $P$ and each band $b$ of
$\hat{M}$:

$\rho^{\sigma}_{P\hat{M}}(i,j,b)=\mbox{corrcoef}\left(P^{\sigma}_{ij},\hat{M}(b)^{\sigma}_{ij}\right)$
(11)

Then we reduce it to its average value $\overline{\rho^{\sigma}_{P\hat{M}}}$
over space and spectral bands. The final index is then defined as

$D_{\rho}\triangleq 1-\overline{\rho^{\sigma}_{P\hat{M}}},$ (12)

such that $D_{\rho}=0$ corresponds to perfect correlation. From a different
point of view, we are studying to what extent a $\sigma{\times}\sigma$ patch
of any band of $\hat{M}$ can be linearly predicted from the corresponding PAN
patch. Therefore, $D_{\rho}$ is strictly related to the matching between the
spatial layouts of $\hat{M}$ and $P$ at a fine scale.

The choice of the scale parameter $\sigma$ is of critical importance, as we
are interested to the exploitation of the complementary information discarded
by the reprojection indexes, i.e. details appearing only at the PAN scale but
not at the MS scale. Therefore, if $R$ is the resolution ratio, by choosing
$\sigma=R$ (equal to 4 for our datasets) we can put the focus just on that
spatial content which is not visible at the MS scale. By doing so, we reduce
the dependence between the proposed spatial ($D_{\rho}$) and spectral
(R-$\mathcal{M}$) distortion indexes, assigning to them clear but well
separated evaluation objectives.

As for $D_{\rm S}$ and $D_{\lambda}^{(\rm K)}$, or its variants, $D_{\rho}$
computed for $\hat{M}$=GT does not reach the zero value, given the slightly
different spatial layout of different bands in natural images and the complex
relationship between the high resolution image and its panchromatic projection
[72]. Nonetheless, we expect good quality pansharpened images to present
relatively small values of $D_{\rho}$. A detailed discussion in this regard
supported by our experimental results will be provided in the following.

##  4 Experimental results and discussion

In this Section we present several experimental analyses. Preliminarily, a
summary of the datasets employed and on the involved methods is provided. The
first experiment focuses on the dependence of the spectral consistency indexes
on the alignments of the MS bands with the PAN. Next, we move to the reduced-
resolution domain to cross-check reference-based and no-reference indexes
thanks to the availability of the ground-truth. Then, it follows an
experimental analysis focused on the assessment of the spatial consistency,
before closing with an overall comparison.

###  4.1 Datasets and methods

Table 1: Bandwidths of the multispectral channels (left-hand side) and Ground Sample Distance (GSD) [m] at Nadir (rightmost column) for WorldView-2 and WorldView-3 images. Sensor | Bandwidths of the MS Channels [nm] | GSD [m]
---|---|---
| Coastal | Red | Blue | Red Edge | Green | Near-IR1 | Yellow |  Near-IR2 | PAN/MS
WV2  | 396-458 | 624-694 | 442-515 | 699-749 | 506-586 | 765-901 | 584-632 | 856-1043  | 0.46 / 1.84
WV3  | 400-450 | 630-690 | 450-510 | 705-745 | 510-580 | 770-895 | 585-625 | 860-1040  | 0.31 / 1.24

The experimental validation relies on 25 methods provided in the benchmark
toolbox [79] belonging to the four main categories recalled in Section 1, CS
(8), MRA (9), VO (3) and ML (4), plus an ideal interpolator (EXP). The dataset
is composed by two WorldView-2 (WV2) and two WorldView-3 (WV3) large images,
courtesy sample products of DigitalGlobe©. 20 $512{\times}512$ tiles were
extracted from the WV2 images (Washington and Stockholm) and 20 from the WV3
images (Adelaide and Mexico City) for the experiments at full resolution.
Likewise, 20+20 $2048{\times}2048$ tiles were extracted and downscaled to size
$512{\times}512$ for the experiments at reduced resolution. Tab. 1 summarizes
the main spectral and spatial characteristic of the WV-2 and WV-3 sensors.

###  4.2 Spectral distortion dependence on PAN-MS misalignment

Table 2: Misregistration impact on $D_{\lambda}$, $D_{\lambda}^{(\rm K)}$, $\widetilde{D}_{\lambda}^{(\rm K)}$ and $D_{\lambda,\rm align}^{(\rm K)}$. Detailed information about the methods can be found in [79]. |  |  $D_{\lambda}$ |  $D_{\lambda}^{(\rm K)}$ |  $\widetilde{D}_{\lambda}^{(\rm K)}$ | $D_{\lambda,\rm align}^{(\rm K)}$
---|---|---|---|---|---
| Aligned dataset  | yes | no  | yes | no  | yes | no  | yes | no
|  EXP (interpolator)  | 0.0000 | 0.0000  | 0.0249 | 0.0336  | 0.0394 | 0.0479  | 0.1238 | 0.0670
CS |  BT-H [44] | 0.0698 | 0.0823  | 0.1231 | 0.0696  | 0.2310 | 0.1324  | 0.0874 | 0.0697
BDSD [26] | 0.0429 | 0.0377  | 0.0989 | 0.0867  | 0.1834 | 0.1561  | 0.1511 | 0.1064
C-BDSD [24] | 0.0514 | 0.0557  | 0.1436 | 0.1195  | 0.2158 | 0.1836  | 0.2346 | 0.1664
BDSD-PC [77] | 0.0141 | 0.0160  | 0.0782 | 0.0896  | 0.1464 | 0.1530  | 0.0705 | 0.0892
GS [37] | 0.0196 | 0.0177  | 0.1494 | 0.0824  | 0.2611 | 0.1414  | 0.0824 | 0.0839
GSA [5] | 0.0576 | 0.0573  | 0.1182 | 0.0604  | 0.2457 | 0.1334  | 0.0760 | 0.0624
C-GSA [63] | 0.0309 | 0.0333  | 0.0929 | 0.0583  | 0.1925 | 0.1240  | 0.0741 | 0.0625
PRACS [17] | 0.0178 | 0.0193  | 0.0728 | 0.0468  | 0.1464 | 0.0889  | 0.0834 | 0.0610
MRA |  AWLP [8] | 0.0332 | 0.0416  | 0.0282 | 0.0273  | 0.0415 | 0.0320  | 0.0920 | 0.0495
MTF-GLP [8] | 0.0679 | 0.0759  | 0.0231 | 0.0195  | 0.0434 | 0.0352  | 0.0912 | 0.0428
MTF-GLP-FS [80] | 0.0544 | 0.0669  | 0.0222 | 0.0199  | 0.0400 | 0.0347  | 0.0944 | 0.0439
MTF-GLP-HPM [8] | 0.0620 | 0.0722  | 0.0310 | 0.0210  | 0.0508 | 0.0376  | 0.0909 | 0.0411
MTF-GLP-HPM-H [44] | 0.1019 | 0.1177  | 0.0270 | 0.0200  | 0.0524 | 0.0401  | 0.0885 | 0.0402
MTF-GLP-HPM-R [81] | 0.0497 | 0.0620  | 0.0272 | 0.0210  | 0.0453 | 0.0362  | 0.0928 | 0.0420
MTF-GLP-CBD [9] | 0.0551 | 0.0657  | 0.0222 | 0.0200  | 0.0402 | 0.0346  | 0.0942 | 0.0441
C-MTF-GLP-CBD [63] | 0.0089 | 0.0375  | 0.0224 | 0.0225  | 0.0360 | 0.0347  | 0.1129 | 0.0495
MF [64] | 0.0585 | 0.0711  | 0.0421 | 0.0281  | 0.0681 | 0.0441  | 0.0733 | 0.0371
VO |  FE-HPM [82] | 0.0579 | 0.0666  | 0.0340 | 0.0210  | 0.0529 | 0.0355  | 0.0784 | 0.0406
SR-D [75] | 0.0225 | 0.0381  | 0.0043 | 0.0064  | 0.0117 | 0.0190  | 0.1254 | 0.0450
TV [56] | 0.0166 | 0.0238  | 0.0225 | 0.0173  | 0.0459 | 0.0373  | 0.0583 | 0.0252
ML |  PNN [45] | 0.0589 | 0.0553  | 0.0740 | 0.0629  | 0.1477 | 0.1307  | 0.1815 | 0.0878
PNN-IDX [45] | 0.0837 | 0.0848  | 0.2671 | 0.0546  | 0.1570 | 0.1005  | 0.3344 | 0.0889
A-PNN [66] | 0.0527 | 0.0636  | 0.0371 | 0.0312  | 0.0773 | 0.0607  | 0.0963 | 0.0522
A-PNN-FT [66] | 0.0196 | 0.0203  | 0.0414 | 0.0308  | 0.0815 | 0.0579  | 0.0830 | 0.0489

The impact of bands misregistration on the quality of the fused products has
been already recognized in the past [12, 31, 90]. Here we propose an ad hoc
experimental analysis to show the robustness of the proposed reprojection
indexes to the data mis-registration. The starting point is a WV3 dataset
composed of ten 2048$\times$2048 tiles extracted from a larger image of
Adelaide (DigitalGlobe© sample product). This dataset presents bands
misalignment that we have corrected to produce a companion aligned dataset. In
Tab. 2 we summarize the average spectral distortion scores for each method for
both datasets. For a handy reading of these numbers, in Fig. 2 we show the
impact of data misregistration on $D_{\lambda}^{(\rm K)}$ and
$\widetilde{D}_{\lambda}^{(\rm K)}$ in differential terms.

Figure 2: Misregistration impact on $D_{\lambda}^{(\rm K)}$ and
$\widetilde{D}_{\lambda}^{(\rm K)}$. Bar levels indicate the variation of the
indexes due to data misragistration.

Each bar indicates the difference between the values of the given indicator
computed on aligned and misaligned datasets, respectively. As it can be
clearly noticed, traditional CS methods such as BT-H, GS, GSA, C-GSA, PRACS,
that by construction provide fused images that are strongly anchored to the
PAN geometry, pay a considerable spectral loss according to $D_{\lambda}^{(\rm
K)}$ or $\widetilde{D}_{\lambda}^{(\rm K)}$. These results are not aligned
with our expectations, as these CS methods are expected to be more robust to
misregistration. The Reader can refer to [12] for a theoretical comparison
between CS and MRA methods in presence of misregistration, with the former
category being superior to the latter.

In Fig. 3, instead, it is compared Khan’s index $D_{\lambda}^{(\rm K)}$ with
its variant $D_{\lambda,\rm align}^{(\rm K)}$ with alignment. Again, bars
indicate their variations due to dataset misalignment.

Figure 3: Misregistration impact on $D_{\lambda}^{(\rm K)}$ and
$D_{\lambda,\rm align}^{(\rm K)}$. Bar levels indicate the variation of the
indexes due to data misragistration.

By inspecting the figure it can be noticed that misregistered data generally
cause an increase of $D_{\lambda,\rm align}^{(\rm K)}$, except for some CS
methods, such as BT-H, GS, GSA, C-GSA and PRACS, which tend to preserve the
PAN geometry operating an intrinsic alignment. On the contrary, other methods,
notably those belonging to the MRA, VO and ML categories, that are oriented to
the spectral preservation but do not operate alignment, register an increase
of the spectral distortion when the metric takes into account the
misalignment.

Concluding, the proposed experiment suggests that Khan’s index with alignment
correction, that is the complement of the proposed reprojection index
R-$Q2^{n}$, is more robust to misalignment or, at least, more consistent with
the theoretical expectation [12], than the original Khan’s index
$D_{\lambda}^{(\rm K)}$ or its variant $\widetilde{D}_{\lambda}^{(\rm K)}$.

###  4.3 Reference vs no-reference indexes cross-checking in the reduced
resolution space

In order to assess the consistency between no-reference and reference-based
indicators, we have designed a set of experiments in the reduced resolution
space. Despite no-reference indexes are conceived to work in the full-
resolution framework, their use on (simulated) reduced resolution datasets
allows us to carry out studies on the correlation between them and objective
error measurements (reference-based indexes) thanks to the availability of
GTs. In particular, for this experiment we have resorted to our WV2
dataset111For the sake of brevity, analogous results obtained on WV3 images
will not be presented. composed of twenty 2048$\times$2048 images at full-
resolution. These images come from two larger images of Washington (13) and
Stockholm (7), respectively, courtesy samples of DigitalGlobe©. Each such
tiles was resized to 512$\times$512 pixels using the usual Wald’s downgrading
protocol. This dataset was already well coregistered, therefore we have
proceeded to create a misregistered counterpart operating a simple
modification of the downgrading process: a 1-pixel shift (in both directions)
has been introduced in the decimation (after LPF) of the MS bands.

Then, 25 pansharpening algorithms provided by the toolbox [79] were run on
each RR sample image generating 500 results, for each of the two datasets
(registered and not), for which all the indexes of interest have been
computed. Eventually, we have obtained hundreds of points in a multi-
dimensional evaluation space, which enable plenty of analyses, with some
dimensions corresponding to the reference-based indexes (SAM, ERGAS and
$Q2^{n}$, SSIM, CMSC), and the remaining ones associated to the proposed
indexes (R-SAM, R-ERGAS, R-$Q2^{n}=1-D_{\lambda,\rm align}^{(\rm K)}$ ,
$D_{\rho}$) and to other state-of-the-art no-reference indexes ($D_{\lambda}$,
$D_{\lambda}^{(\rm K)}$, $\widetilde{D}_{\lambda}^{(\rm K)}$, $D_{\rm S}$,
QLR1, QLR2). In particular, by construction we expect to observe a good
correlation between Khan’s index variants and $Q2^{n}$. Therefore we start
from the scatter plots shown in Fig. 4 with such variants, in turn, vs
$Q2^{n}$.

Figure 4: From left to right $\widetilde{D}_{\lambda}^{(\rm K)}$,
$D_{\lambda}^{(\rm K)}$ and $D_{\lambda,\rm align}^{(\rm K)}=1-$R-$Q2^{n}$ vs
$Q2^{n}$ on the WV2 RR dataset. Both the “aligned” (top) and “mis-aligned”
(bottom) dataset cases are shown. Each marker is associated to a single
pansharpening result on a $512\times 512$ tile. Black markers correspond to
the ground-truth. Light gray ones (EXP) correspond to a simple interpolation.
The remaining clusters correspond to the different algorithms grouped by
categories. Figure 5: From left to right QLR1 vs SSIM and QLR2 vs CMSC on the
WorldView-2 RR dataset. Both the “aligned” (top) and “mis-aligned” (bottom)
dataset cases are shown. Each marker is associated to a single pansharpening
result on a $512\times 512$ tile. Black markers correspond to the ground-
truth. Light gray ones (EXP) correspond to a simple interpolation. The
remaining clusters correspond to the different algorithms grouped by
categories.

Both the “aligned” (top) and the “mis-aligned” (bottom) datasets are
considered. For the aligned case, by visual inspection we can appreciate that
$D_{\lambda}^{(\rm K)}$ correlates better than $\widetilde{D}_{\lambda}^{(\rm
K)}$ with $Q2^{n}$. Besides, $D_{\lambda,\rm align}^{(\rm K)}$ behaves just
like $D_{\lambda}^{(\rm K)}$ because no alignment is operated. For both
$D_{\lambda}^{(\rm K)}$ and $D_{\lambda,\rm align}^{(\rm K)}$, the GTs (black
dots), for which $Q2^{n}$ is always 1, get the ideal value (0) of spectral
distortion.222Rarely, the alignment process (based on the maximization of the
correlation) embedded in the computation of $D_{\lambda,\rm align}^{(\rm K)}$
fails for some spectral band giving raise to non-zero results for the GT even
if the test image is aligned. Besides, $\widetilde{D}_{\lambda}^{(\rm K)}$ is
not minimized by the GT, coherently with its definition (Eq. 6) based on the
comparison between the smoothed GT and the upscaled MS. It is also worth to
notice that the correlation degree between $Q2^{n}$ and the different variants
of spectral distortion grows when assessed by category (by colors), supporting
the idea that no-reference indexes are more reliable for intra-class
assessment [79]. Moving to mis-aligned dataset (bottom scatters), it can be
clearly recognized a bias for both $\widetilde{D}_{\lambda}^{(\rm K)}$ and
$D_{\lambda}^{(\rm K)}$, which do not occurs for $D_{\lambda,\rm align}^{(\rm
K)}$, represented by the shift of the GT scores which no longer get the ideal
value. More in general, the distortion scores in terms of
$\widetilde{D}_{\lambda}^{(\rm K)}$ and $D_{\lambda}^{(\rm K)}$ register a
degradation, with CS methods (blue) much more penalized than others. This last
observation gives a further support the interpretation provided above about
the results of Fig. 2 and Fig. 3. A similar behavior is also registered other
state-of-the-art indexes. For example, in Fig.5 the score scatters in the
QLR1-SSIM (left) and QLR2-CMSC (right) planes, with registered (top) and
misaligned (bottom) datasets.

Let us now leave the registration problem out considering aligned datasets
only and looking at the relationship between the spectral consistency indexes
and the three most used reference-based indexes, i.e., SAM, ERGAS and
$Q2^{n}$, with the help of Fig. 6. From top to bottom we can see how different
compared spectral consistency indexes agrees with the three (column-wise
distributed) reference-based indexes.

Figure 6: No-reference vs reference-based indexes scores on the aligned WV2 RR
dataset. From left to right, SAM, ERGAS and $Q2^{n}$ are selected as
reference-based indexes ($x$-axis). From top to bottom, $D_{\lambda}^{(\rm
K)}$, QLR1, QLR2 and R-$\mathcal{M}$ are chosen as no-reference indexes
($y$-axis).

From the top row scatters it clearly appears that $D_{\lambda}^{(\rm K)}$
agrees with SAM and ERGAS much lesser than with $Q2^{n}$. This because
$D_{\lambda}^{(\rm K)}$ is based on the same $Q2^{n}$ index but also because
SAM and ERGAS encode a different concept of quality. Similar considerations
apply for QLR1 and QLR2 who, as well as $Q2^{n}$, are both based on the
comparison among local statistics. For these reasons, we believe that in
addition to R-$Q2^{n}$ it makes sense to also provide R-SAM and R-ERGAS for a
more comprehensive evaluation of pansharpening methods. On the bottom row, we
show all three ($\mathcal{M}$, R-$\mathcal{M}$) scatter plots, which speak in
favor of a good level of agreement between each objective index $\mathcal{M}$
and the reprojected counterpart R-$\mathcal{M}$.

Beside the qualitative interpretation of the score scatters, we can quantify
the level of agreement among the reference-based indexes and the compared no-
reference indexes in terms of correlation coefficients. These are shown in
Tab. 3 in the usual matrix form, for both the aligned (a) and misaligned (b)
dateset. As expected the reprojected indexes show a relatively high
correlation with the non-reprojected counterpart, both for aligned and
misaligned datasets. $D_{\lambda}^{(\rm K)}$ and
$\widetilde{D}_{\lambda}^{(\rm K)}$ also correlate well with $Q2^{n}$, but
only in the aligned case. Likewise, QLR1 and QLR2 also correlate well with
different reference-based indexes in ideal conditions but register a drop on
misaligned data. It is also worth to remark that GT scores have been discarded
to not penalize excessively the competitors on the misaligned dataset (see GT
score distribution in Fig. 4-5)

Table 3: Correlation coefficient among different indexes (some have been
complemented for a uniform polarization) obtained on the aligned (a) and
misaligned (b) RR WV2 datasets. Outlier and GT scores have been discarded.
---
(a)
(b)

###  4.4 A qualitative assessment of the proposed spatial distortion index

While objective synthesis quality indexes such as $Q2^{n}$ and ERGAS account
for both spectral and spatial inaccuracies, their reprojected versions clearly
limit their scope to the spectral component as they are computed on low-pass
versions of the fused products ignoring image “details”. It is therefore
necessary to complement these indexes with some spatial quality indicator.

The assessment of the spatial quality of a pansharpened image in the full-
resolution framework is very difficult and somewhat controversial, lacking a
degradation model to describe the relationship between the full-resolution
spatial-spectral datacube (fused image) and the panchromatic image. In fact,
while the spectral degradation can be reasonably modeled through the sensor
MTF, allowing for a spectral consistency check, the spatial degradation cannot
be modeled through a simple global weighted band average [72]. Indeed, in
addition to the obvious spatial dependency of the spectral mixing process that
provides the PAN image, there is also a mismatch between the PAN spectral
coverage and the MS coverage [72]. This means that there could be details
“seen” by the PAN but not by the virtual full-resolution MS counterpart, and
vice versa. This is the origin of an ill-posedness of the pansharpening
problem, mostly residing in the spatial reconstruction, which makes the
spatial consistency assessment subjective to some extent. On the basis of this
observation, we provide here an interpretation of some sample results through
visual inspection, comparing the proposed index $D_{\rho}$ with the state-of-
the-art spatial distortion index $D_{\rm S}$. Of course, given the subjective
nature of this kind of analysis, we will focus on clearly visible phenomena,
leaving to the Reader the final say based on his/her visual perception of more
subtle patterns, if any. In particular, we propose two experiments: an
“horizontal” comparison among the pansharpening toolbox methods and a
“vertical” comparison where a single machine learning method is optimized
using the proposed $D_{\rho}$ (varying the related scale parameter $\sigma$)
jointly with a term controlling the spectral consistency.

Let us start with the first experiment. Fig. 7 shows some FR pansharpening
results for crops extracted from a single tile of the WV3 Adelaide
image.333For visualization purposes, hereinafter all displayed crops extracted
from multispectral images are obtained combining the red, green and blue
channels (see Tab. 1). The PAN component, used as a reference, is shown in the
middle of different groups of results. In the top half part of the figure, the
top-4 solutions according to $D_{\rm S}$ (left) and the top-4 according to
$D_{\rho}$ (right) are shown with $D_{\rm S}$ and $D_{\rho}$ computed on the
whole tile. Similarly, in the bottom half part, the top-4 results according to
QHR and NIQE are shown with related scores. It appears clearly that images
selected according to the $D_{\rho}$ index ensure a better agreement with the
reference in terms of spatial layout and, more in general, a better quality,
with sharp contours, accurate textures, and the lack of annoying patterns such
as those present in some top-$D_{\rm S}$ and top-NIQE images. It is
interesting to observe that, among the top-4 $D_{\rm S}$ results, the most
convincing one from visual inspection corresponds also to the relative lowest
$D_{\rho}$ (AWLP). It is also worth to observe a certain degree of agreement
between $D_{\rho}$ and QHR. Similar phenomena are observed on all other tiles.

---
Figure 7: Pansharpening results on crops from a full-resolution WV3 Adelaide
tile. The reference PAN is in the middle column. In the top half the four best
results in terms of $D_{\rm S}$ and $D_{\rho}$ are on its left and right,
respectively, with best scores (in red) closer to the reference. In the bottom
half are shown the top results according to QHR (leftward) and NIQE
(rightward). Best scores are highlighted in red; second, third and fourth
bests are in blue.

In the previous experiment we have set the scale parameter $\sigma=R=4$
according to the above discussed theoretical motivations (Section 3.2). To
gain insight into the effectiveness of this choice we have designed an
additional ad hoc experiment to provide pansharpening results with controlled
spectral and spatial qualities, with the latter quantified through $D_{\rho}$,
configurable using different scale settings. This is achieved by leveraging on
a CNN model working in target-adaptive modality [18] and using a combination
of the wanted consistency measures as a loss. In this context, the choice of
the specific CNN model is not critical as we work in adaptive modality, by
running unsupervised tuning iterations on each test image until the loss terms
reach the desired level (overfitting). In particular, the optimized loss is
defined as

$\mathcal{L}=\max(\|\hat{M}_{\downarrow a}^{\rm
lp}-M\|_{1},\mathcal{L}_{\lambda}^{*})+\beta\max(D_{\rho}^{(\sigma)},D_{\rho}^{*}),$
(13)

where $\mathcal{L}_{\lambda}^{*}$ and $D_{\rho}^{*}$ are two prefixed target
levels for the spectral ($\|\hat{M}_{\downarrow a}^{\rm lp}-M\|_{1}$) and
spatial ($D_{\rho}^{(\sigma)}$) quality indicators, respectively. In practice,
the network parameters are pushed to overfit the test image so that the
corresponding loss reaches the target quality
$(\mathcal{L}_{\lambda}^{*},D_{\rho}^{*})$. These two threshold values could
be ideally set to zero, but this would lead to extremely long tuning and may
also generate instability because of a conflicting interaction between the two
loss terms. In particular, we have set $\mathcal{L}_{\lambda}^{*}=0.015$,
quite a low value considering the dynamic range of the pixels values, and
$D_{\rho}^{*}=0.1$, which is also quite low according to our experiments. By
doing so, each test requires a different number of tuning iterations. We have
therefore oversized the number (experimentally set to 5000) of iterations to
ensure convergence for all. This process is repeated for several choices of
the scale parameter $\sigma$ ranging from 2 to 64. In Fig. 8, we show some
crops from the WV3 dataset. These sample results reflect a general behavior
that we have observed in a wide range of experiments, that is a relatively
good response in the range of $\sigma\in[4,8]$. For smaller ($\sigma=2$) or
larger ($\sigma>8$) scale values noisy patterns arise. These are particularly
noticeable on roofs and roads for the selected samples. The above observations
provide an experimental validation of the choice $\sigma=R=4$ proposed in this
work on the basis of the theoretical motivations discussed in Section 3.2.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  |  |  |  |  |
|  |  |  |  |  |  |
MS | PAN  | $\sigma=2$ | $\sigma=4$ | $\sigma=8$ | $\sigma=16$ | $\sigma=32$ | $\sigma=64$
Figure 8: Pansharpening results with controlled spectral ($\ell_{1}$-norm) and
spatial ($D_{\rho}$) distortions. From left to right: MS and PAN, followed by
the outputs optimized using $\sigma\in\\{2,4,8,16,32,64\\}$.

The present experiment gives us the opportunity to also make some
considerations about the computational load related to the proposed indexes.
In general, in the context of quality assessment, the computation time is not
a critical issue as it is carried out offline. However, with the advent of
deep learning, researchers started to use such quality indexes as loss
functions for training purposes. In this regard, it is worth to distinguish
pixel-based indexes (e.g., SAM, ERGAS, $\ell_{\alpha}$-norms), that are
relatively light to compute, from other indexes that involve local statistics
computed at a certain scale for each location (e.g., $Q2^{n}$, $D_{\rho}$,
$D_{\lambda}^{(\rm K)}$). When using the latter as loss the training may
slowdown. In the particular case of $D_{\rho}$, that has been included in the
loss function to finetune the sample CNN employed in the experiment, we have
actually registered a moderate impact on the training time. In fact, using a
training batch composed of a single 2048$\times$2048 image, the time
consumption per iteration, without or with the additional $D_{\rho}$ loss
component (second term in Eq. 13), shifts from 1.27s to 1.6s on a NVIDIA P6000
GPU.

###  4.5 Comparative results

To conclude, we present here an overall comparison among the available
pansharpening methods provided by the toolbox [79], using the proposed quality
indicators and supported by the visual inspection of some sample results. In
Fig. 9 and Fig. 10 we gather the average numerical results obtained on our WV3
and WV2 datasets, respectively.

Figure 9: Full-resolution assessment on the WV3 dataset. R-$Q2^{n}$ is
replaced by its complement $D_{\lambda,\rm align}^{(\rm K)}$ for the sake of
readability (0 is the ideal value for all). For details on methods and related
categories (CS, MRA, VO, ML) refer to Tab. 2. Figure 10: Full-resolution
assessment on the WV2 dataset. R-$Q2^{n}$ is replaced by its complement
$D_{\lambda,\rm align}^{(\rm K)}$ for the sake of readability (0 is the ideal
value for all). For details on methods and related categories (CS, MRA, VO,
ML) refer to Tab. 2.

As expected, the numerical results clearly show a good level of agreement
among the three reprojection error indexes, all essentially linked to the
spectral consistency. However, some exceptions can be observed, particularly
for the WV2 dataset (Fig. 10) where some CS methods register a performance
loss in terms of $D_{\lambda,\rm align}^{(\rm K)}$ not observed on R-SAM and
R-ERGAS. We also recognize a good agreement with some literature findings,
particularly on WV3 (Fig. 9), such as the spectral accuracy gap between MRA
and CS methods [78] or the competitiveness of some MRA, VO and ML solutions.
Moreover, for the ML solutions, it is worth to notice a performance gap moving
from one dataset (WV3) to another (WV2). Indeed, such variability is not
unusual for data-driven approaches that can suffer generalization limits when
the training dataset is not sufficiently representative. In this particular
situation, it could likely be the case that the WV2 test images used are too
different from the images used for the training of the involved ML methods.

To gain insight into the effectiveness of the proposed indexes let us give a
look to some sample results for a complementary subjective analysis. In Fig.
11 we show some clips from a single WV3 tile of the Adelaide dataset. On the
leftmost column are gathered the PAN and MS input components on two
consecutive rows. Then, the corresponding pansharpening results of the best
performing solutions according to R-$Q2^{n}$ are shown in decreasing order
from left to right, next to the PAN. Next to the MS, we also show the
“reprojection” error map, that is the difference between the input MS and the
reprojection (LPF plus decimation) of the output. On the bottom lines are
gathered the R-$Q2^{n}$ and $D_{\rho}$ scores. Therefore, the spectral quality
of the results decreases left to right. This can also be appreciated through
the inspection of the reprojection error maps. Besides, the best methods from
the spatial perspective ($D_{\rho}$) follow a different ordering. This
partially reflects a tradeoff between spectral and spatial features.
Particularly interesting is the case of the method TV which scores first in
terms of R-$Q2^{n}$ (actually it is the best according to R-SAM and R-ERGAS,
as well) but shows the worst $D_{\rho}$ value, 0.215, corresponding to a 78.5%
average correlation between the PAN and the pansharpened bands. The impact of
this low correlation level is clearly visible in the pansharpening results of
TV which show underlying noisy patterns. Such patterns disappear with the
reprojection explaining why the reprojection indexes do not worsen. The other
methods achieve much lower values of $D_{\rho}$, ranging from 0.084 to 0.115
(around 90% of correlation), to which correspond to a higher coherence between
the spatial features of the results and the PAN, easily appreciable through
visual inspection. For completeness, in Fig. 12 we show analogous results for
a WV2 tile, which basically confirms the same considerations made above for
WV3.

|  |  |  |  |  |  |
---|---|---|---|---|---|---|---
PAN  | TV | MF | …-GLP-HPM | AWLP | …-HPM-H | FE-HPM | MTF-GLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
|  |  |  |  |  |  |
PAN  | TV | MF | …-GLP-HPM | AWLP | …-HPM-H | FE-HPM | MTF-GLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
|  |  |  |  |  |  |
PAN  | TV | MF | …-GLP-HPM | AWLP | …-HPM-H | FE-HPM | MTF-GLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
| R-$Q2^{n}$=0.950 | R-$Q2^{n}$=0.945 | R-$Q2^{n}$=0.939 | R-$Q2^{n}$=0.937 | R-$Q2^{n}$=0.937 | R-$Q2^{n}$=0.937 | R-$Q2^{n}$=0.936
|  $D_{\rho}$=0.215 |  $D_{\rho}$=0.115 |  $D_{\rho}$=0.090 |  $D_{\rho}$=0.105 |  $D_{\rho}$=0.096 |  $D_{\rho}$=0.114 |  $D_{\rho}$=0.084
Figure 11: Sample crops from a WV3 sample image (Adelaide) and related full-resolution pansharpening. For each crop, in addition to the PAN and MS components (left-most column), the best performing solutions according to R-$Q2^{n}$ are shown from left to right in decreasing order, next to the PAN. For each method, both the R-$Q2^{n}$ and $D_{\rho}$ scores are reported on the bottom lines of the figure. Moreover, next to the MS, for each pansharpening result the corresponding reprojection error map is shown. |  |  |  |  |  |  |
---|---|---|---|---|---|---|---
PAN  | TV | MF | FE-HPM | SR-D | …-GLP-HPM | MTF-GLP | AWLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
|  |  |  |  |  |  |
PAN  | TV | MF | FE-HPM | SR-D | …-GLP-HPM | MTF-GLP | AWLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
|  |  |  |  |  |  |
PAN  | TV | MF | FE-HPM | SR-D | …-GLP-HPM | MTF-GLP | AWLP
|  |  |  |  |  |  |
MS  | Reprojection error maps: defference between the MS and the reprojected pansharpening
| R-$Q2^{n}$=0.986 | R-$Q2^{n}$=0.977 | R-$Q2^{n}$=0.976 | R-$Q2^{n}$=0.974 | R-$Q2^{n}$=0.974 | R-$Q2^{n}$=0.974 | R-$Q2^{n}$=0.973
|  $D_{\rho}$=0.227 |  $D_{\rho}$=0.102 |  $D_{\rho}$=0.095 |  $D_{\rho}$=0.238 |  $D_{\rho}$=0.079 |  $D_{\rho}$=0.076 |  $D_{\rho}$=0.097
Figure 12: Sample crops from a WV2 sample image (Washington) and related full-
resolution pansharpening. For each crop, in addition to the PAN and MS
components (left-most column), the best performing solutions according to
R-$Q2^{n}$ are shown from left to right in decreasing order, next to the PAN.
For each method, both the R-$Q2^{n}$ and $D_{\rho}$ scores are reported on the
bottom lines of the figure. Moreover, next to the MS, for each pansharpening
result the corresponding reprojection error map is shown.

##  5 Conclusions

To cope with the limitations of the reference-based reduced-resolution
procedures for the quality assessment of pansharpening methods, in this work
we propose a new full-resolution no-reference evaluation framework. By
following Wald’s protocol [86], the full-resolution assessment must be carried
out checking the “consistency” of the fused products with the MS and PAN input
components rather than the “synthesis” capacity which would require the
availability of GTs. In particular, inspired by Khan’s index [51], we have
proposed the use of reprojection-based indexes with embedded alignment to
handle mis-registered datasets for the assessment of the spectral consistency
between the pansharpened image and the input MS. Besides, the spatial
consistency between the fused image and the PAN is quantified by averaging,
spatially and spectrally, the fine-scale local correlation of individual
super-resolved bands with the high-resolution PAN.

A key qualifying aspect of the proposed indexes is the absence of any
resolution downgrading of the input data, which frees the assessment from the
effect of scale-dependent phenomena. Experiments on reduced resolution
datasets show that the reprojection indexes are reliable predictors of image
quality as quantified by reference-based indexes supporting their use in the
full-resolution domain. On the other hand, experiments on full-resolution data
make clear that the local correlation-based index provides indications on
image quality that largely agree with the judgement of human experts. The
proposed approach can be readily generalized to fusion tasks other than
pansharpening such as, for example, the combination of low-resolution
hyperspectral and high-resolution multispectral images.

On the other hand, the User must also be aware of some limitations. In
particular, the reprojection requires the knowledge of an accurate estimation
of the sensor MTF for a correct low-pass filtering. A wrong estimation would
lead to inaccurate spectral consistency assessment, a problem shared with
preexisting solutions. Moreover, the proposed correlation distortion index is
based on the assumption that the PAN correlates well with all spectral bands.
Such a hypothesis is globally acceptable but there can be rare cases,
spatially and spectrally well localized, for which it results too strong.

In order to ensure reproducible research, the code for the proposed indexes
can be found at https://github.com/matciotola/fr-pansh-eval-tool/.

## References

  * [1] B. Aiazzi, L. Alparone, S. Baronti, R. Carlà, A. Garzelli, and L. Santurri.  Full scale assessment of pansharpening methods and data products.  In Proc. SPIE, volume 9244, pages 1–12, 2014.
  * [2] B. Aiazzi, L. Alparone, S. Baronti, and A. Garzelli.  Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis.  IEEE Trans. Geosci. Remote Sens., 40(10):2300–2312, Oct 2002.
  * [3] B. Aiazzi, L. Alparone, S. Baronti, A. Garzelli, and M. Selva.  An MTF-based spectral distortion minimizing model for pan-sharpening of very high resolution multispectral images of urban areas.  In GRSS/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, pages 90–94, May 2003.
  * [4] B Aiazzi, L Alparone, S Baronti, A Garzelli, and M Selva.  Mtf-tailored multiscale fusion of high-resolution ms and pan imagery.  Photogramm. Eng. Rem. S., 72(5):591–596, 2006.
  * [5] B. Aiazzi, S. Baronti, and M. Selva.  Improving component substitution pansharpening through multivariate regression of MS+Pan data.  IEEE Trans. Geosci. Remote Sens., 45(10):3230–3239, Oct 2007.
  * [6] L. Alparone, B. Aiazzi, S. Baronti, A. Garzelli, F. Nencini, and M. Selva.  Multispectral and panchromatic data fusion assessment without reference.  Photogramm. Eng. Rem. S., 74(2):193–200, 2008.
  * [7] L. Alparone, S. Baronti, A. Garzelli, and F. Nencini.  A global quality measurement of pan-sharpened multispectral imagery.  Geoscience and Remote Sensing Letters, IEEE, 1(4):313–317, Oct 2004\.
  * [8] L. Alparone, A. Garzelli, and G. Vivone.  Intersensor statistical matching for pansharpening: Theoretical issues and practical solutions.  IEEE Transactions on Geoscience and Remote Sensing, 55(8):4682–4695, 2017.
  * [9] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L.M. Bruce.  Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S Data-Fusion Contest.  IEEE Trans. Geosci. Remote Sens., 45(10):3012–3021, Oct 2007.
  * [10] Luciano Alparone, Andrea Garzelli, and Gemine Vivone.  Spatial consistency for full-scale assessment of pansharpening.  In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 5132–5134. IEEE, 2018.
  * [11] A. Azarang and H. Ghassemian.  A new pansharpening method using multi resolution analysis framework and deep neural networks.  In 2017 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA), pages 1–6, April 2017.
  * [12] Stefano Baronti, Bruno Aiazzi, Massimo Selva, Andrea Garzelli, and Luciano Alparone.  A theoretical analysis of the effects of aliasing and misregistration on pansharpened imagery.  IEEE Journal of Selected Topics in Signal Processing, 5(3):446–453, 2011.
  * [13] P. Benedetti, D. Ienco, R. Gaetano, K. Ose, R. G. Pensa, and S. Dupuy.  $m^{3}\text{Fusion}$: A deep learning architecture for multiscale multimodal multitemporal satellite data fusion.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(12):4939–4949, Dec 2018.
  * [14] Roberto Carla, Leonardo Santurri, Bruno Aiazzi, and Stefano Baronti.  Full-scale assessment of pansharpening through polynomial fitting of multiscale measurements.  IEEE Transactions on Geoscience and Remote Sensing, 53(12):6344–6355, 2015.
  * [15] P.S. Chavez and A.W. Kwarteng.  Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis.  Photogrammetric Engineering and Remote Sensing, 55(3):339 – 348, 1989.
  * [16] M. Cheng, C. Wang, and J. Li.  Sparse representation based pansharpening using trained dictionary.  IEEE Geoscience and Remote Sensing Letters, 11(1):293–297, 2014\.
  * [17] J. Choi, K. Yu, and Y. Kim.  A new adaptive component-substitution-based satellite image fusion by using partial replacement.  IEEE Trans. Geosci. Remote Sens., 49(1):295–309, Jan 2011.
  * [18] Matteo Ciotola, Sergio Vitale, Antonio Mazza, Giovanni Poggi, and Giuseppe Scarpa.  Pansharpening by convolutional neural networks in the full resolution framework.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–1, 2022\.
  * [19] Ingrid Daubechies.  Orthonormal bases of compactly supported wavelets ii. variations on a theme.  SIAM Journal on Mathematical Analysis, 24(2):499–519, 1993.
  * [20] C. Dong, C.C. Loy, K. He, and X. Tang.  Image super-resolution using deep convolutional networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2):295–307, Feb 2016.
  * [21] Wenqian Dong, Shaoxiong Hou, Song Xiao, Jiahui Qu, Qian Du, and Yunsong Li.  Generative dual-adversarial network with spectral fidelity and spatial enhancement for hyperspectral pansharpening.  IEEE Transactions on Neural Networks and Learning Systems, 2021\.
  * [22] Wenqian Dong, Tongzhen Zhang, Jiahui Qu, Song Xiao, Jie Liang, and Yunsong Li.  Laplacian pyramid dense network for hyperspectral pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [23] D. Fasbender, J. Radoux, and P. Bogaert.  Bayesian data fusion for adaptable image pansharpening.  IEEE Trans. Geosci. Remote Sens., 46(6):1847–1857, June 2008.
  * [24] A. Garzelli.  Pansharpening of multispectral images based on nonlocal parameter optimization.  IEEE Trans. Geosci. Remote Sens., 53(4):2096–2107, April 2015.
  * [25] A. Garzelli and F. Nencini.  Hypercomplex quality assessment of multi/hyperspectral images.  Geoscience and Remote Sensing Letters, IEEE, 6(4):662–665, Oct 2009\.
  * [26] A. Garzelli, F. Nencini, and L. Capobianco.  Optimal MMSE pan sharpening of very high resolution multispectral images.  IEEE Trans. Geosci. Remote Sens., 46(1):228–236, Jan 2008.
  * [27] Alan R Gillespie, Anne B Kahle, and Richard E Walker.  Color enhancement of highly correlated images. II. Channel ratio and “chromaticity” transformation techniques.  Remote Sensing of Environment, 22(3):343–365, 1987.
  * [28] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980–2988, Oct 2017.
  * [29] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu.  An augmented linear mixing model to address spectral variability for hyperspectral unmixing.  IEEE Transactions on Image Processing, 28(4):1923–1938, April 2019\.
  * [30] D. Hong, N. Yokoya, J. Chanussot, and X. X. Zhu.  Cospace: Common subspace learning from hyperspectral-multispectral correspondences.  IEEE Transactions on Geoscience and Remote Sensing, 57(7):4349–4359, July 2019.
  * [31] Linhai Jing, Qiuming Cheng, Huadong Guo, and Qizhong Lin.  Image misalignment caused by decimation in image fusion evaluation.  International Journal of Remote Sensing, 33(16):4967–4981, 2012\.
  * [32] M.M. Khan, J. Chanussot, L. Condat, and A. Montanvert.  Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique.  IEEE Geoscience and Remote Sensing Letters, 5(1):98–102, Jan 2008\.
  * [33] Elham Kordi Ghasrodashti and Nabin Sharma.  Hyperspectral image classification using an extended auto-encoder method.  Signal Processing: Image Communication, 92:116111, 2021.
  * [34] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, pages 1106–1114, 2012.
  * [35] Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz.  The spectral image processing system (sips)—interactive visualization and analysis of imaging spectrometer data.  Remote sensing of environment, 44(2-3):145–163, 1993.
  * [36] Chiman Kwan, Bence Budavari, Alan C. Bovik, and Giovanni Marchisio.  Blind quality assessment of fused worldview-3 images by using the combinations of pansharpening and hypersharpening paradigms.  IEEE Geoscience and Remote Sensing Letters, 14(10):1835–1839, 2017\.
  * [37] C.A. Laben. and B.V. Brower.  Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening.  U.S. Patent 6011875, 2000., 2000.
  * [38] C. Lanaras, E. Baltsavias, and K. Schindler.  Hyperspectral super-resolution by coupled spectral unmixing.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 3586–3594, Dec 2015.
  * [39] Fahad Lateef and Yassine Ruichek.  Survey on semantic segmentation using deep learning techniques.  Neurocomputing, 338:321 – 348, 2019.
  * [40] J. Lee and C. Lee.  Fast and efficient panchromatic sharpening.  IEEE Trans. Geosci. Remote Sens., 48(1):155–163, Jan 2010.
  * [41] S. Li, H. Yin, and L. Fang.  Remote sensing image fusion via sparse representations over learned dictionaries.  IEEE Trans. Geosci. Remote Sens., 51(9):4779–4789, Sept 2013.
  * [42] Shutao Li and Bin Yang.  A new pan-sharpening method using a compressed sensing technique.  IEEE Trans. Geosci. Remote Sens., 49(2):738–746, Feb 2011.
  * [43] X. Liu, Y. Wang, and Q. Liu.  Psgan: A generative adversarial network for remote sensing image pan-sharpening.  In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 873–877, Oct 2018.
  * [44] S. Lolli, L. Alparone, A. Garzelli, and G. Vivone.  Haze correction for contrast-based multispectral pansharpening.  IEEE Geoscience and Remote Sensing Letters, 14(12):2255–2259, 2017\.
  * [45] G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa.  Pansharpening by convolutional neural networks.  Remote Sens., 8(7):594, 2016.
  * [46] G. Masi, D. Cozzolino, L. Verdoliva, and G. Scarpa.  Cnn-based pansharpening of multi-resolution remote-sensing images.  In Joint Urban Remote Sensing Event 2017, Dubai, 6–8 March 2017\.
  * [47] Antonio Mazza, Francescopaolo Sica, Paola Rizzoli, and Giuseppe Scarpa.  Tandem-x forest mapping using convolutional neural networks.  Remote Sensing, 11(24), 2019.
  * [48] X. Meng, H. Shen, H. Li, Q. Yuan, H. Zhang, and L. Zhang.  Improving the spatial resolution of hyperspectral image using panchromatic and multispectral images: An integrated method.  In WHISPERS, June 2015.
  * [49] Xiangchao Meng, Kedi Bao, Jinfang Shu, Bingzhong Zhou, Feng Shao, Weiwei Sun, and Shutao Li.  A blind full-resolution quality evaluation method for pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, 60:1–16, 2022\.
  * [50] Anish Mittal, Rajiv Soundararajan, and Alan C Bovik.  Making a “completely blind” image quality analyzer.  IEEE Signal processing letters, 20(3):209–212, 2012.
  * [51] M.M.Khan, L.Alparone, and J.Chanussot.  Pansharpening quality assessment using the modulation transfer functions of instruments.  IEEE Trans. Geosci. Remote Sens., 47(11):3880–3891, 2009.
  * [52] Curtis K Munechika, James S Warnick, Carl Salvaggio, and John R Schott.  Resolution enhancement of multispectral image data to improve classification accuracy.  Photogrammetric Engineering and Remote Sensing, 59(1):67–72, 1993\.
  * [53] J. Nunez, X. Otazu, O. Fors, A. Prades, V. Pala, and R. Arbiol.  Multiresolution-based image fusion with additive wavelet decomposition.  IEEE Trans. Geosci. Remote Sens., 37(3):1204–1211, May 1999.
  * [54] X. Otazu, M. Gonzalez-Audicana, O. Fors, and J. Nunez.  Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods.  IEEE Trans. Geosci. Remote Sens., 43(10):2376–2385, Oct 2005.
  * [55] F. Palsson, J. R. Sveinsson, M. O. Ulfarsson, and J. A. Benediktsson.  Model-based fusion of multi- and hyperspectral images using pca and wavelets.  IEEE Transactions on Geoscience and Remote Sensing, 53(5):2652–2663, 2015.
  * [56] F. Palsson, J.R. Sveinsson, and M.O. Ulfarsson.  A new pansharpening algorithm based on total variation.  Geoscience and Remote Sensing Letters, IEEE, 11(1):318–322, Jan 2014.
  * [57] F. Palsson, M. O. Ulfarsson, and J. R. Sveinsson.  Model-based reduced-rank pansharpening.  IEEE Geoscience and Remote Sensing Letters, 17(4):656–660, 2020\.
  * [58] Gintautas Palubinskas.  Quality assessment of pan-sharpening methods.  In 2014 IEEE Geoscience and Remote Sensing Symposium, pages 2526–2529. IEEE, 2014.
  * [59] Gintautas Palubinskas.  Joint quality measure for evaluation of pansharpening accuracy.  Remote Sensing, 7(7):9292–9310, 2015.
  * [60] T. Ranchin and L. Wald.  Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation.  Photogrammetric engineering and remote sensing, 66(1):49–61, 2000\.
  * [61] Thierry Ranchin and Lucien Wald.  The wavelet transform for the analysis of remotely sensed images.  International Journal of Remote Sensing, 14(3):615–619, 1993.
  * [62] Y. Rao, L. He, and J. Zhu.  A residual convolutional neural network for pan-shaprening.  In 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP), pages 1–4, May 2017.
  * [63] R. Restaino, M. Dalla Mura, G. Vivone, and J. Chanussot.  Context-adaptive pansharpening based on image segmentation.  IEEE Transactions on Geoscience and Remote Sensing, 55(2):753–766, Feb 2017.
  * [64] R. Restaino, G. Vivone, M. Dalla Mura, and J. Chanussot.  Fusion of multispectral and panchromatic images based on morphological operators.  IEEE Transactions on Image Processing, 25(6):2882–2895, 2016.
  * [65] G. Scarpa, M. Gargiulo, A. Mazza, and R. Gaetano.  A CNN-Based Fusion Method for Feature Extraction from Sentinel Data.  Remote Sensing, 10(2):236, 2018.
  * [66] G. Scarpa, S. Vitale, and D. Cozzolino.  Target-adaptive CNN-based pansharpening.  IEEE Transactions on Geoscience and Remote Sensing, 56(9):5443–5457, Sep. 2018.
  * [67] V. P. Shah, N. H. Younan, and R. L. King.  An efficient pan-sharpening method via a combined adaptive pca approach and contourlets.  IEEE Transactions on Geoscience and Remote Sensing, 46(5):1323–1335, May 2008.
  * [68] Z. Shao and J. Cai.  Remote sensing image fusion with deep convolutional neural network.  IEEE J. Sel. Topics Appl. Earth Observ., 11(5):1656–1669, May 2018\.
  * [69] H. Shen, X. Meng, and L. Zhang.  An integrated framework for the spatio-temporal-spectral fusion of remote sensing images.  IEEE Trans. Geosci. Remote Sens., 54(12):7135–7148, Dec 2016.
  * [70] V.K. Shettigara.  A generalized component substitution technique for spatial enhancement of multispectral images using a higher resolution data set.  Photogramm. Eng. Remote Sens., 58(5):561–567, 1992.
  * [71] Timothy P. Szczykutowicz, Sean D. Rose, and Alexander Kitt.  Sub pixel resolution using spectral-spatial encoding in x-ray imaging.  PLOS ONE, 16:1–15, 10 2021.
  * [72] Claire Thomas, Thierry Ranchin, Lucien Wald, and Jocelyn Chanussot.  Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics.  IEEE Transactions on Geoscience and Remote Sensing, 46(5):1301–1312, 2008.
  * [73] T.-M. Tu, P. S. Huang, C.-L. Hung, and C.-P. Chang.  A fast intensity hue-saturation fusion technique with spectral adjustment for ikonos imagery.  IEEE Geoscience and Remote Sensing Letters, 1(4):309–312, 2004\.
  * [74] Te-Ming Tu, Shun-Chi Su, Hsuen-Chyun Shyu, and Ping S. Huang.  A new look at ihs-like image fusion methods.  Information Fusion, 2(3):177 – 186, 2001.
  * [75] M. R. Vicinanza, R. Restaino, G. Vivone, M. Dalla Mura, and J. Chanussot.  A pansharpening method based on the sparse representation of injected details.  IEEE Geosci. Remote Sens. Lett., 12(1):180–184, Jan 2015.
  * [76] S. Vitale, G. Ferraioli, and G. Scarpa.  A cnn-based model for pansharpening of worldview-3 images.  In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pages 5108–5111, 2018.
  * [77] G. Vivone.  Robust band-dependent spatial-detail approaches for panchromatic sharpening.  IEEE Transactions on Geoscience and Remote Sensing, 57(9):6421–6433, 2019.
  * [78] G. Vivone, L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, G. A. Licciardi, R. Restaino, and L. Wald.  A critical comparison among pansharpening algorithms.  IEEE Trans. Geosci. Remote Sens., 53(5):2565–2586, May 2015.
  * [79] G. Vivone, M. Dalla Mura, A. Garzelli, R. Restaino, G. Scarpa, M. O. Ulfarsson, L. Alparone, and J. Chanussot.  A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods.  IEEE Geoscience and Remote Sensing Magazine, 2020.
  * [80] G. Vivone, R. Restaino, and J. Chanussot.  Full scale regression-based injection coefficients for panchromatic sharpening.  IEEE Transactions on Image Processing, 27(7):3418–3431, 2018.
  * [81] G. Vivone, R. Restaino, and J. Chanussot.  A regression-based high-pass modulation pansharpening approach.  IEEE Transactions on Geoscience and Remote Sensing, 56(2):984–996, 2018.
  * [82] G. Vivone, M. Simões, M. Dalla Mura, R. Restaino, J. M. Bioucas-Dias, G. A. Licciardi, and J. Chanussot.  Pansharpening based on semiblind deconvolution.  IEEE Transactions on Geoscience and Remote Sensing, 53(4):1997–2010, 2015.
  * [83] Gemine Vivone, Paolo Addesso, and Jocelyn Chanussot.  A combiner-based full resolution quality assessment index for pansharpening.  IEEE Geoscience and Remote Sensing Letters, 16(3):437–441, 2018\.
  * [84] Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot.  A bayesian procedure for full-resolution quality assessment of pansharpened products.  IEEE Transactions on Geoscience and Remote Sensing, 56(8):4820–4834, 2018.
  * [85] Lucien Wald.  Quality of high resolution synthesised images: Is there a simple criterion?  In Third conference" Fusion of Earth data: merging point measurements, raster maps and remotely sensed images", pages 99–103. SEE/URISCA, 2000.
  * [86] Lucien Wald, Thierry Ranchin, and Marc Mangolini.  Fusion of satellite images of different spatial resolution: Assessing the quality of resulting images.  Photogramm. Eng. Remote Sensing, pages 691–699, 1997.
  * [87] Zhou Wang and A. C. Bovik.  A universal image quality index.  IEEE Signal Processing Letters, 9(3):81–84, 2002.
  * [88] Y. Wei and Q. Yuan.  Deep residual learning for remote sensed imagery pansharpening.  In 2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP), pages 1–4, May 2017.
  * [89] Y. Wei, Q. Yuan, H. Shen, and L. Zhang.  Boosting the accuracy of multispectral image pansharpening by learning a deep residual network.  IEEE Geoscience and Remote Sensing Letters, 14(10):1795–1799, Oct 2017.
  * [90] Qizhi Xu, Yun Zhang, and Bo Li.  Recent advances in pansharpening and key problems in applications.  International Journal of Image and Data Fusion, 5(3):175–195, 2014\.
  * [91] J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding, and J. Paisley.  Pannet: A deep network architecture for pan-sharpening.  In ICCV, Oct. 2017.
  * [92] N. Yokoya, T. Yairi, and A. Iwasaki.  Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion.  IEEE Transactions on Geoscience and Remote Sensing, 50(2):528–537, Feb 2012.
  * [93] Q. Yuan, Y. Wei, X. Meng, H. Shen, and L. Zhang.  A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11:978–989, Mar. 2018.
  * [94] L. Zhang, H. Shen, W. Gong, and H. Zhang.  Adjustable model-based fusion method for multispectral and panchromatic images.  IEEE Trans. Syst. Man Cybern. B Cybern., 42(6):1693–1704, Dec 2012\.
  * [95] Y. Zhang, C. Liu, M. Sun, and Y. Ou.  Pan-sharpening using an efficient bidirectional pyramid network.  IEEE Trans. Geosci. Remote Sens., 57(8):5549–5563, Aug. 2019.
  * [96] Z. Zhao, P. Zheng, S. Xu, and X. Wu.  Object detection with deep learning: A review.  IEEE Transactions on Neural Networks and Learning Systems, 30(11):3212–3232, Nov 2019.
  * [97] S. Zhong, Y. Zhang, Y. Chen, and D. Wu.  Combining component substitution and multiresolution analysis: A novel generalized bdsd pansharpening algorithm.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 10(6):2867–2875, June 2017.
  * [98] J. Zhou, D. L. Civco, and J. A. Silander.  A wavelet transform method to merge landsat tm and spot panchromatic data.  International Journal of Remote Sensing, 19(4):743–757, 1998.
  * [99] Hongyan Zhu, Aoife Gowen, Hailin Feng, Keping Yu, and Jun-Li Xu.  Deep spectral-spatial features of near infrared hyperspectral images for pixel-wise classification of food products.  Sensors, 20(18), 2020.
  * [100] X. X. Zhu, C. Grohnfeldt, and R. Bamler.  Exploiting joint sparsity for pansharpening: The j-sparsefi algorithm.  IEEE Transactions on Geoscience and Remote Sensing, 54(5):2664–2681, May 2016.
  * [101] X.X. Zhu and R. Bamler.  A sparse image fusion algorithm with application to pan-sharpening.  IEEE Trans. Geosci. Remote Sens., 51(5):2827–2836, May 2013.