# Strain-driven valley states and phase transitions in Janus VSiGeN4 monolayer

Pengyu Liu  Institute for Computational Materials Science, Joint Center for
Theoretical Physics (JCTP), School of Physics and Electronics, Henan
University, Kaifeng, 475004, China  Siyuan Liu  Institute for Computational
Materials Science, Joint Center for Theoretical Physics (JCTP), School of
Physics and Electronics, Henan University, Kaifeng, 475004, China  Minglei Jia
Institute for Computational Materials Science, Joint Center for Theoretical
Physics (JCTP), School of Physics and Electronics, Henan University, Kaifeng,
475004, China  Huabing Yin  Institute for Computational Materials Science,
Joint Center for Theoretical Physics (JCTP), School of Physics and
Electronics, Henan University, Kaifeng, 475004, China  Guangbiao Zhang
Institute for Computational Materials Science, Joint Center for Theoretical
Physics (JCTP), School of Physics and Electronics, Henan University, Kaifeng,
475004, China  Fengzhu Ren<EMAIL_ADDRESS>Institute for Computational
Materials Science, Joint Center for Theoretical Physics (JCTP), School of
Physics and Electronics, Henan University, Kaifeng, 475004, China  Bing Wang
<EMAIL_ADDRESS>Institute for Computational Materials Science, Joint Center for
Theoretical Physics (JCTP), School of Physics and Electronics, Henan
University, Kaifeng, 475004, China  Chang Liu<EMAIL_ADDRESS>Institute
for Computational Materials Science, Joint Center for Theoretical Physics
(JCTP), School of Physics and Electronics, Henan University, Kaifeng, 475004,
China

###### Abstract

The interplay between topology and valley degree of freedom has attracted much
interest because it can realize new phenomena and applications. Here, based on
first-principles calculations, we demonstrate intrinsically valley-polarized
quantum anomalous Hall effect in monolayer ferrovalley material: Janus
VSiGeN4, of which the edge states are chiral-spin-valley locking. Besides, a
small tensile or compressive strain can drive phase transition in the material
from valley-polarized quantum anomalous Hall state to half-valley-metal state.
With the increase of the strain, the material turns into ferrovalley
semiconductor with valley anomalous Hall effect. The origin of phase
transition is sequent band inversion of V d orbital at K valley. Moreover, we
find that phase transition causes the sign reversal of Berry curvature and
induces different polarized light absorption in different valley states. Our
work provides an ideal material platform for practical applications and
experimental exploration of the interplay between topology, spintronics, and
valleytronics.

##  I Introduction

The valley, local maximum or local minimum of electronic band structure in
momentum space, provides valley degree of freedom to encode and manipulate
information analogous to charge and spin Gunawan  _et al._ (2006); Xiao  _et
al._ (2007); Schaibley  _et al._ (2016), the related field is called
valleytronics. For utilizing the valley index as an information carrier, the
key is how to break the energy degeneracy of the valley in the valleytronics
material, which can create carries imbalance between different valleys to be
detected Rycerz  _et al._ (2007). The experimental discovery of two-
dimensional (2D) transition metal dichalcogenides (TMDs) has greatly promoted
the development of valleytronics Aivazian  _et al._ (2014); Zeng  _et al._
(2012); Mak  _et al._ (2012); Cao  _et al._ (2012); Xiao  _et al._ (2012); Wu
_et al._ (2013); Li  _et al._ (2014); MacNeill  _et al._ (2015); Srivastava
_et al._ (2015); Sallen  _et al._ (2012); Kioseoglou  _et al._ (2012); Guan
_et al._ (2021); Guan and Luo (2020). It is found that optical pumping can
produce valley polarization in MoS2 based on the valley-contrasting optical
selection rules Zeng  _et al._ (2012); Mak  _et al._ (2012), a finding that
can be applied to valley optoelectronic devices Withers  _et al._ (2015); Lee
_et al._ (2014). Unfortunately, due to the short lifetime of the carries, the
valley will depolarize quickly and cause information loss. On the other hand,
some experiments reported that Zeeman effect of external magnetic field can
break the energy degeneracy of valleys in TMD Li  _et al._ (2014); MacNeill
_et al._ (2015). However, the efficiency of magnetic field is low ($0.1\sim
0.2$ meV/T), thereby it is difficult to preserve a suitable valley splitting
value by magnetic field in electronic equipment. Besides, it was found that
magnetic doping and magnetic proximity effect can induce valley polarization
Xu  _et al._ (2018); Norden  _et al._ (2019); Peng  _et al._ (2018).
Especially, a larger than 300 meV valley splitting in MoTe2 can be induced by
magnetic substrate of EuO Qi  _et al._ (2015). Yet, the magnetic substrate
submerges the valley physics of host materials, and magnetic doping usually
increases impurity scatterings between different valleys. Valley polarization
induced by above ways is volatile. Thus, it is important to find intrinsic
valley polarization.

Recently, valleytronic material have been extended to 2D ferromagnetic system
named ferrovalley Tong  _et al._ (2016); Feng  _et al._ (2021); He  _et al._
(2021); Cheng  _et al._ (2021); Cui  _et al._ (2021); Zhao  _et al._ (2019);
Jiang  _et al._ (2021); Guo  _et al._ (2021); Huan  _et al._ (2021); Zang  _et
al._ (2021); Li  _et al._ (2021); Hu  _et al._ (2020); Zhao  _et al._ (2021);
Liu  _et al._ (2020a). When both time-reversal and inversion symmetry are
broken, ferrovalley materials can produce spontaneous valley polarization and
realize anomalous valley Hall effect (AVHE) Tong  _et al._ (2016); Cheng  _et
al._ (2021), and these properties can be well applied to data process or data
storage. Importantly, out-of-plane magnetization is a necessary condition for
achieving spontaneous valley polarization Zhao  _et al._ (2019); Zang  _et
al._ (2021); Du  _et al._ (2020). However, most of the discovered ferrovalley
materials have in-plane magnetization Cui  _et al._ (2021); Zhao  _et al._
(2019); Guo  _et al._ (2021); Liu  _et al._ (2017), and thus require difficult
modulation of the magnetic easy axis. On the other hand, some novel quantum
states were discovered in ferrovalley materials, such as topological state and
half-valley-metal (HVM) state Huan  _et al._ (2021); Hu  _et al._ (2020). The
multiple valley states not only enrich valley degree of freedom and quantum
states but also bring intriguing physical properties, such as valley-polarized
quantum anomalous effect (VQAHE) and selective absorption of polarized light
Feng  _et al._ (2021); Li  _et al._ (2021); Hu  _et al._ (2020). Though highly
valuable, ferrovalley material with both modulate multiple valley states and
out-of-plane magnetization is quite rare.

In this work, we predict a ferrovalley material of Janus VSiGeN4 monolayer
based on density-functional theory (DFT). We find that this monolayer can not
only maintain valley characteristics, but also has large valley polarization
and strain-induced phase transition. Our results show that monolayer VSiGeN4
is a ferromagnetic semiconductor, and it has out-of-plane magnetization with a
Curie temperature of 113K. Interestingly, the material possesses a VQAHE with
chiral-spin-valley locking edge states. Besides, it becomes HVM with 100$\%$
valley polarization under tiny tensile strain of 0.1$\%$ or compressive strain
of 0.32$\%$. By continuing to increase the strain, the material will transform
into ferrovalley semiconductor (FVS), which shows an AVHE. Moreover, the
polarized light absorption and the sign of berry curvature located in the K$+$
and K$-$ valley can also be modulated by applying strain. Our work thus
enriches materials with valley-related multiple Hall effect and provides a
platform for research valleytronics, spintronics, and topology.

Figure 1: (a) and (b) Top and side views of ML VSiGeN4, respectively. (c)
Phonon dispersion spectrum and (d) energy variation during the 6 ps AIMD
simulation at 300K. Insets in (d) display the initial and final structures of
ML VSiGeN4 in the MD simulation.

##  II Computational Methods

All the first-principles calculations were performed using the projector-
augmented wave (PAW) method Blöchl (1994), which is embedded in the Vienna ab
initio Simulation Package (VASP) Kresse and Furthmüller (1996). The exchange
and correlation interaction is described by the Predew-Burke-Ernzerhof (PBE)
functional with the general gradient approximation (GGA) Perdew  _et al._
(1996). The spin-orbit coupling (SOC) effect is included in the calculations.
We use $\Gamma$-centered k-meshes of 15$\times$15$\times$1 for the Brillouin-
zone (BZ) sampling and 500 eV cutoff energy for the plane wave basis. The
convergence criterion for energy and force is chosen as 10-6 eV and 0.01 eV/Å,
respectively. A vacuum region of 20 Å is taken to eliminate the interactions
between periodic layers. The GGA+U method is adopted to describe the strong
correlated correction of V-3d orbitals and the U parameter is chosen as 3 eV
Feng  _et al._ (2021); Cui  _et al._ (2021); Liu  _et al._ (2020a). The ab
initio molecular dynamic (AIMD) simulation is performed by using a
4$\times$4$\times$1 supercell to determin thermal stability. Phonon spectrum
is calculated by using the density functional perturbation theory (DFPT) with
a 4$\times$4$\times$1 supercell. The edge states and Chern number were
calculated by using the WANNIER90 package Pizzi  _et al._ (2020) and
WANNIERTOOLS package Wu  _et al._ (2018). The Monte Carlo (MC) simulations are
performed on a 2D 16$\times$16 supercell based on the standard Metropolis-
Hasting algorithm, and we use 640000 sweeps to get reasonable result.

##  III STRUCTURE STABILITIES AND MAGNETISM

The Janus monolayer (ML) VSiGeN4 has a triangular lattice structure with space
group P3M1, and the optimized lattice constant is a $=$ b $=$ 2.96 Å. As shown
in Fig. 1(a) and Fig. 1(b), the lattice has no space inversion symmetry, and
the structure is built up by septuple atomic layers in the sequence of N-Ge-N-
V-N-Si-N, which can be viewed as VN2 layer sandwiched by a Ge-N and Si-N
layer. To confirm the stability of structure, the phonon spectra and molecular
dynamic (MD) simulation are shown in Fig. 1(c) and Fig. 1(d), respectively.
The absence of imaginary frequencies in phonon spectra confirms the dynamics
stability. After 6 ps MD simulation at 300K, no destruction is observed in ML
VSiGeN4, which indicate the thermal stability under 300K. Besides, we
calculated the formation energy of ML VSiGeN4:
$E_{f}=(E_{total}-E_{Si}-E_{V}-E_{Ge}-4E_{N})/n$. Where Etotal represent the
total energy of ML VSiGeN4 in the primitive cell, ESi, EV, EGe, and EN is the
energy per atom of Si, V, Ge and N in their most stable phases, respectively,
and n is the number of atoms in the primitive cell. The calculated value of Ef
is about $-$0.602 eV/atom, implying that ML VSiGeN4 has a good thermodynamical
stability against the elemental phases. Therefore, we expect that ML VSiGeN4
can be synthesized experimentally, for example by substituting atoms in ML
MoSi2N4 Hong  _et al._ (2020).

Next, the magnetic properties of ML VSiGeN4 is investigated. We calculate the
total energies of different magnetic states: ferromagnetic (FM), nonmagnetic
(NM), antiferromagnetic (AFM1 and 120∘ AFM) structures, which are shown in
Fig. S1. The results show that the total energy of FM structure is the lowest,
76 meV per unit cell lower than the AFM1 state, 351 meV per unit cell lower
than NM state. Besides, the spin configuration of 120∘ AFM is unstable and
transforms into FM state, which is similar to 2H-VSSe and LaBrI Jiang  _et
al._ (2021); Zhang  _et al._ (2019). Thus, FM state is the ground state, which
can be interpreted by the Goodenough-Kanamori-Anderson (GKA) rules Wu  _et
al._ (2021); Goodenough (1955); Anderson (1959); Kanamori (1960). In ML
VSiGeN4, the V-N-V bonding angle is 91.8∘, close to 90∘. According to the GKA
rules, such a structure configuration favors FM coupling. To determine the
easy magnetization axis, we calculated magnetic anisotropy energy (MAE), which
is defined as E${}_{M}=$ E${}_{[100]}-E_{[001]}$, where E[100] and E[001]
represent the total energy of system with the spin orientation of V atoms
along out-of-plane and in-plane, respectivley. The EM is 72 $\mu$eV per unit
cell, which means that ML VSiGeN4 is out-of-plane magnetization. Such
magnetization direction is necessary to realize the spontaneous valley
polarization.

To study the magnetic stability of ML VSiGeN4, we estimated the Curie
temperature (TC) by using the classical Metropolis MC simulations based on the
Heisenberg model: $H=-\sum_{<ij>}J_{ij}S_{i}\cdot S_{j}-D{(S_{iz})^{2}}$.
Where $S_{i}$ is the normalized spin vector on site $i$, $J_{ij}$ is the
exchange interaction strength between sites $i$ and $j$, $D$ is the single-
site MAE, $S_{iz}$ represents component of $S$ along $z$ orientation, and
$\lvert{S}\rvert=\frac{1}{2}$ for V in 2D VSiGeN4. Because the moment of the V
atom is around 1 $\mu_{B}$. With $J=$ 51.49 meV and $D=$ 72 $\mu$eV, the
result of MC simulation shows that the value of TC is about 113 K (Fig.
S1(d)), which is much higher than CrI3 (45 K) and Cr2Ge2Te6 bilayer (30 K)
Huang  _et al._ (2017); Gong  _et al._ (2017).

Figure 2: The spin polarized band structures of ML VSiGeN4 (a) without SOC;
(b) and (c) with SOC for magnetic moment of V along the positive and negative
z direction, respectively. (d) Orbital-resolved spin polarized band structure
with SOC in ML material. Spin projections along the out-of-plane direction are
indicated by red (positive direction, spin-up) and blue (negative direction,
spin-down) line for VSiGeN4 in (b) and (c).

##  IV ELECTRONIC STRUCTURES AND STRAIN-DRIVEN VALLEY STATES

The spin polarized band structure of the ML VSiGeN4 without considering SOC is
shown in Fig. 2(a). It can be seen that spin-down and spin-up channels split
significantly, and there is only spin-up states around the Fermi level (EF).
The material is a semiconductor with a small direct band gap of 22.7 meV. Both
the valence band maximum (VBM) and the conduction band minimum (CBM) locate at
the high symmetry point K. The degenerate valleys at the K$+$ and K$-$ point
are nonequivalent due to the break of the inversion symmetry. Therefore, ML
VSiGeN4 is a valleytronic material. Similar to the previously reported
ferrovalley materials, when considering both SOC and magnetic exchange
interaction, ML VSiGeN4 spontaneously enables a valley polarization without
any additional tuning. As shown in Fig. 2(b), the SOC lifts the energy
degeneracy of K$+$ and K$-$ valley (the energy of K$-$ valley is higher than
K$+$ valley for both valence band and conduction band) with valley
polarization of 60.4 meV (18.2 meV) at conduction (valence) band. This
polarization value is larger than previously those of reported materials, such
as VSi2P4 (49.4 meV) Feng  _et al._ (2021), LaBr2 (33meV) Zhao  _et al._
(2019), Cr2Se3 (18.7meV) He  _et al._ (2021) and TiVI6 (22meV) Du  _et al._
(2020). More importantly, the influence of irrelevant bands for valley
properties is minimum in the energy range of ($-0.2\sim 1$ eV). The linear
band dispersion around K$-$ and K$+$ valley show a high Fermi velocity of
about $0.31\times{10}^{6}~{}{ms}^{-1}$, which has the same order of magnitude
as graphene. The valley polarization can be tuned by an external magnetic
field. As shown in Fig. 2(c), the valley polarization and spin state are
flipped by reversing the magnetization of V atoms, so that all K valley states
now occupy the spin-down channel and K$-$ valley state has a lower energy than
K$+$ valley. The tunable large valley polarization, large Fermi velocity, and
the clean single spin state around EF, endow VSiGeN4 with more advantages than
most valleytronics materials in practical applications.

The different values of valley polarization between K$+$ and K$-$ valley can
be attributed to the different orbital contributions. As shown in Fig. 2(d),
at K$+$ valley, the CBM is contributed by $d_{z^{2}}$ orbital of V and the VBM
is contributed by $d_{x^{2}-y^{2}}+d_{xy}$ orbital of V, while the situation
at K$-$ valley is reversed. Since the magnetic quantum number ($m_{z}$) of
$d_{z^{2}}$ orbital is 0, the basis functions composed of this orbital is the
same at K$+$ and the K$-$ point. While basis functions composed of
$d_{x^{2}-y^{2}}$ and $d_{xy}$ orbital is different at K$+$ and the K$-$
point, which can be chosen as:
$\left|\psi\right\rangle$=$\frac{1}{\sqrt{2}}(\left|d_{x^{2}+y^{2}}\right\rangle+i\tau\left|d_{xy}\right\rangle)$.
Here $\tau=\pm 1$ indicates the valley index at the $K\pm$ point Feng  _et
al._ (2021). Therefore, introducing SOC into a system with neither time-
reversal symmetry nor space-reversal symmetry will result in different energy
eigenvalues at K$+$ point and K$-$ point with different basis functions,
leading to the occurrence of valley polarization.

Figure 3: (a) and (b) The distribution of Berry curvature along the high
symmetry path and along the whole BZ in strain-free ML VSiGeN4, respectively.
(c) and (d) The edge state for ML VSiGeN4 with magnetic moment along positive
and negative $z$ direction, respectively.

The opposite orbital order for the K$-$ and K$+$ valleys imply that ML VSiGeN4
has topological properties, such as VQAH effect. This anomalous transport
phenomenon is related to the Berry curvature $\Omega(k)$. For a 2D systems,
$\Omega(k)$ only has $z$ component $\Omega_{z}(k)$. It can be calculated by Li
_et al._ (2020):

$\displaystyle\Omega_{z}(k)=-\sum_{n}\sum_{n\neq
n^{{}^{\prime}}}f_{n}\frac{2Im\langle\psi_{nk}|\upsilon_{x}|\psi_{n^{{}^{\prime}}k}\rangle\langle\psi_{n^{{}^{\prime}}k}|\upsilon_{y}|\psi_{nk}\rangle}{(E_{n}-E_{n}^{{}^{\prime}})^{2}},$

where $n$ and $n^{{}^{\prime}}$ are band indexes; $f_{n}$ is the equilibrium
Fermi-Dirac distribution function for the $n$-th band at a k point;
$\upsilon_{x(y)}$ is the velocity operator along the $x(y)$ direction;

Figure 4: (a) The evolution of orbital-projected band with SOC under different
biaxial strain, respectively. (b) and (c) The edge state for ML VSiGeN4 under
$\varepsilon=-1\%$ and $\varepsilon=1\%$, respectively. (d) The diagram of
topological phase transitions and the variation of the band gaps at K valley
with different strain.

$k$ is the wave vector; $\psi_{nk}$ is the Bloch wave function with eigenvalue
$E_{n}$ and $E_{n^{{}^{\prime}}}$.

In Fig. 3(a) and Fig. 3(b), we show the $\Omega_{z}(k)$ distribution of ML
VSiGeN4 along the high symmetry path and contour map in the 2D Brillouin zone
(BZ), respectively. One clearly observes that the values of $\Omega_{z}(k)$
are all zero except the region around K$+$ and K$-$ valleys. More importantly,
the sign is same and the magnitudes are different from two valleys. With such
distribution of $\Omega_{z}(k)$, the material may have a nonzero Chern number
and edge states protected by the nontrivial topological property, which result
in a VQAH effect. We have evaluated the Chern number (C) of ML VSiGeN4 via
formula: $C=\frac{1}{2\pi}\int_{BZ}^{\ }{d^{2}k{\Omega}_{z}(k)}$. The result
shows that the Chern number is 1. Then, we calculated the edge states as shown
in Fig. 3(c), it can be seen that a topological nontrivial chiral edge state
connecting conduction bands (K$+$ valley) and valence bands (K$-$ valley) is
clearly visible. Due to CBM and VBM all dominated by spin-up states, the edge
state is also spin-up with 100$\%$ spin polarization. Similar to the reversal
of valley polarization under external magnetic field, when the magnetization
of V is reversed, the edge state will change accordingly. As shown in Fig.
3(d), it becomes spin-down state and connect conduction bands (K$+$ valley)
and valence bands (K$-$ valley) with opposite chiral. Besides, the conductive
edge state of spin-up in the global band gap mainly locate at K$+$ valley, the
location of edge state change from K$+$ valley to K$-$ valley when spin
flipping, which revealed a behavior of the chiral-spin-valley locking for the
edge state Huan  _et al._ (2021). Thus, above properties indicate that ML
VSiGeN4 features an intrinsically VQAH effect.

The electronic properties of low-dimensional materials can be generally tuned
by strain Guan  _et al._ (2017); Liu  _et al._ (2020b, 2019, 2021a, 2021b,
2022). Here, we introduce biaxial strain in ML VSiGeN4 and study the effect of
strain on the band structure. The biaxial strain strength is defined as
$\varepsilon=\frac{b-b_{0}}{b_{0}}$, where $b$ and $b_{0}$ are the strained
and unstrained lattice constant, respectively. The result is shown in Fig.
4(a) and Fig. S2. It shows that the band gap is obviously changed under
different strain. Specially, when $\varepsilon=-$0.32$\%$ or 0.1$\%$, the
material transforms into HVM state with 100$\%$ valley polarization and
100$\%$ spin polarization. In such a state, one valley is metallic and the
other one is semiconductor with same spin-up state. The gapless valley shows a
Dirac cone-like linear dispersion with a Fermi velocity of $3.58\times
10^{5}~{}{ms}^{-1}$. Changing the strain can open a gap, and the HVM state is
thus disappeared. As the strain increases, the band gap of the two valleys
also increases gradually, whereas the large valley polarization always remains
(see Fig. 4(d)). Specially, the valley polarization only appears at valence
band for $-$1$\%$ strained VSiGeN4, while in the case of 1$\%$ strain, only
the conduction band has different energy for K$+$ and K$-$ valley. This
phenomenon also originates from the change of orbital composition, as
discussed earlier. Only $d_{x^{2}-y^{2}}+d_{xy}$ orbital has different energy
eigenvalues at different valley, and this orbital constitute the valence band
for $\varepsilon=-1\%$ and the conduction band for $\varepsilon=1\%$,
respectively. Thus, leading to different valley polarization behaviors.

Figure 5: (a) and (b) The distribution of Berry curvature along the high
symmetry path and along the whole BZ under $\varepsilon=-1\%$ and
$\varepsilon=1\%$, respectively. (c) Schematic diagram of multiple valley Hall
effect in ML VSiGeN4 tuned by strain. The holes and electrons are denoted by
blue “$+$” and orange “$-$” symbol, respectively. Upward arrows in orange
color refer to the spin-up carriers. Red frame represents edge state.

As shown in Fig. 4(a), from $\varepsilon=-$1$\%$ to $\varepsilon=$ 1$\%$, the
orbital composition of the band at two valleys is reversed. This band
inversion occurs sequentially in the K$-$ ($\varepsilon=-$0.32$\%$) and K$+$
valleys ($\varepsilon=$ 0.1$\%$) via two HVM states, implying the occurrence
of topological phase transition. We calculated edge state of the material
under $\varepsilon=\pm$1$\%$, respectively. As shown in Fig. 4(b) and Fig.
4(c), one clearly observes that an edge state connecting conduction bands
($\varepsilon=-1\%$) or valence bands ($\varepsilon=1\%$). Therefore, the
system transforms from topological nontrivial state (VQAH state) to trivial
state (FVS) under strain. In order to understand the division of different
phases more intuitively, we plot a phase diagram with the variation of the
band gaps at two valleys for ML VSiGeN4. As shown in Fig. 4(d), the band gaps
change linearly with strain at two valleys, and the band gap at each valley
has a process of closing and reopening, where the gapless states in the
process correspond to the two HVM states. During this process, a topological
phase transition from VQAH state to FVS occurs. Therefore, small strain can
induce phase transition and effectively tune the electronic structure along
with the valley state. Besides, we also calculated energy difference between
different magnetic states under strain (Fig. S3), which reveal that FM state
is still the ground state under different strain.

For FVS systems, there exist nonzero Berry curvature with different values at
K$+$ and K$-$ valley. As shown in Fig. 5(a) and Fig. 5(b), when
$\varepsilon=-1\%$, the $\Omega_{z}(k)$ have different magnitudes with
opposite signs at two valleys, which revealing the typical valley contrasting
characteristic in ML VSiGeN4. Due to the topological phase transition, both
the sign and magnitude of $\Omega_{z}(k)$ are modified (see Fig. 3(a)). So,
when $\varepsilon=$ 1$\%$, the sign of $\Omega_{z}(k)$ at K$+$ and K$-$ valley
is opposite to the case of $\varepsilon=-$1$\%$. As a result, the sign and
magnitude of $\Omega_{z}(k)$ for different valleys can be changed by applying
a small strain. The tunable sign of Berry curvature has been reported by
reversing ferroelectric polarization and magnetization Tong  _et al._ (2016);
Liu  _et al._ (2020a). Here, we propose a convenient method to modulate the
Berry curvature.

With a nonzero $\Omega_{z}(k)$, the electrons will acquire an anomalous
transverse velocity $\upsilon_{n}(k)$ under an in-plane longitudinal electric
field $E$, where
$\upsilon_{n}(k)=\frac{\partial\varepsilon_{n}(k)}{\hbar\partial
k}-\frac{e}{\hbar}E\times\mathrm{\Omega}_{n}(k)$ Xiao  _et al._ (2010), and
induce AVHE. Fig. 5(c) displays multiple valley Hall effects in ML VSiGeN4
which can be tuned by applying strain. For strain-free ML VSiGeN4, the VQAHE
with one spin-polarized edge state can be presented. The edge state is chiral-
spin-valley locking, which means that when the magnetization direction is
reversed, the chiral, spin, and valley of the edge state will also flip. For
FVS state under compressive strain (larger than $-$0.32$\%$), when shifting
the Fermi level between the K$+$ and K$-$ valleys in the valence band (Fig. 4
and Fig. S2), the spin-up holes at K$-$ valley flow to right side of the
sample under an external in-plane electric field $E$, thus the AVHE occurs.
For FVS state under tensile strain (larger than 0.1$\%$), valley polarization
occurs at conduction band. In this case, the spin-up electrons from the K$+$
valley accumulated at the left side of the sample under electric field $E$
With spin-up carriers are only accumulated at one side of the sample, the Hall
voltage can be detected. Besides, when reverse the magnetization orientation,
the conductive valley is changed (from K$+$(K$-$) valley to K$-$(K$+$)
valley), and because of the spin flipping, the direction of the lateral
movement of the carriers changes accordingly, and thus the sign of Hall
voltage changes. Therefore, the valley degree of freedom can be selectively
manipulated in ML VSiGeN4 by applying strain and magnetic field.

Figure 6: The BZ colour-coded by the degree of circular polarization $\eta(k)$
of MLVSiGeN4 under different strain, respectively. $\pm$1 representative
completely absorption left- ($\sigma_{+}$) and right-handed ($\sigma_{-}$)
polarized light, respectively.

Except for valley-related Hall effect, the different polarized light
absorption is also an important topic of valleytronics and valley physics. The
chirality of different valleys in the same material is usually different, and
one chirality corresponds to one polarized light absorption. The change in
sign of $\Omega_{z}(k)$ under strain implies the change in valley chirality.
We calculated valley-selective circular dichroism of ML VSiGeN4 under
different strain as displayed in Fig. 6. The degree of circular polarization
$\eta(k)=\frac{{|M_{+}(k)|}^{2}-{|M_{-}(k)|}^{2}}{{|M_{+}(k)|}^{2}+{|M_{-}(k)|}^{2}}$
Cao  _et al._ (2012) indicates the difference between the absorption of right-
handed and left-handed polarized lights ($\sigma_{\mp}$) at each k point, and
$M_{\pm}(k)$ is the transition matrix of circular polarization. One clearly
observes that both the K$\pm$ valleys mainly absorbs $\sigma_{-}$ in strain-
free ML VSiGeN4. This selective absorption of polarized light can be used in
optoelectronic devices such as filter. For two HVM states, the material still
mainly absorbs $\sigma_{-}$, but only one of the two valleys has light
absorption because the other valley is gapless. Differently, the absorption
occurs at K$+$ valley for HVM1 state ($\varepsilon=-0.32\%$) and K$-$ valley
for HVM2 state ($\varepsilon=0.1\%$). Under a larger stain, the phase
transition will occurs, and the valley chirality will change. As shown in Fig.
6, when $\varepsilon=\pm 1\%$, the two valleys absorb different polarized
lights, and for a particular valley, opposite polarized light is absorbed
under tensile and compressive strain. As a result, strain can also induce
change of valley chirality, enabling the manipulation of polarized light
absorption at different valleys.

##  V CONCLUSIONS

In summary, combining first principles calculations with Wannier-function-
based tight-binding model, we demonstrate the structural stability,
ferromagnetism, valley-dependent properties, topological properties and
strain-induced phase transition of ML VSiGeN4. The results show that ML
VSiGeN4 is a ferromagnetic semiconductor with out-of-plane magnetization and a
$T_{C}$ of 113 K. Simultaneously, it intrinsically exhibits a VQAHE with
100$\%$ spin polarization edge state and spontaneous valley polarization. By
applying tensile strain of 0.1$\%$ and compressive strain of 0.32$\%$, the HVM
state with 100$\%$ valley polarization can be acquired. The FVS state can be
achieved by further increasing the strain, and it has an AVHE that can be
modulated by flipping the magnetization direction. The strain-tunable
polarized light absorption at different valleys can be applied to the next
generation photoelectric device. Our work provides a good material platform
for the interplay between valleytronics, spintronics, and topology.

###### Acknowledgements.

This work was financially supported by the National Natural Science Foundation
of China (No. 11904079, No. 12104130, No. 12047517), the Natural Science
Foundation of Henan (No. 202300410069), the China Postdoctoral Science
Foundation (No. 2020M682274, No. 2020TQ0089) and the Foundation of Henan
Educational Committee (No. 22A140015). Thanks to Dr. Shan Guan from Institute
of Semiconductors in Chinese Academy of Sciences and Dr. Botao Fu from College
of Physics and Electronic Engineering in Sichuan Normal University for their
helpful discussions. The authors acknowledge Beijng PARATERA Tech CO.,Ltd. for
providing HPC resources that have contributed to the research results reported
within this paper.

## References

  * Gunawan  _et al._ (2006) O. Gunawan, Y. P. Shkolnikov, K. Vakili, T. Gokmen, E. P. De Poortere, and M. Shayegan, “Valley susceptibility of an interacting two-dimensional electron system,” Phys. Rev. Lett. 97, 186404 (2006).
  * Xiao  _et al._ (2007) D. Xiao, W. Yao, and Q. Niu, “Valley-contrasting physics in graphene: Magnetic moment and topological transport,” Phys. Rev. Lett. 99, 236809 (2007).
  * Schaibley  _et al._ (2016) J. Schaibley, H. Yu, G. Clark, P. Rivera, J. Ross, K. Seyler, W. Yao, and X. Xu, “Valleytronics in $2\mathrm{D}$ materials,” Nat. Rev. Mater 1, 16055 (2016).
  * Rycerz  _et al._ (2007) A. Rycerz, J. worzydło, and C. W. J. Beenakker, “Valley filter and valley valve in graphene,” Nat. Phys. 3, 172 (2007).
  * Aivazian  _et al._ (2014) G. Aivazian, Z. Gong, A. Jones, R.L. Chu, J. Yan, D. G. Mandrus, C. Zhang, D. Cobden, W. Yao, and X. Xu, “Magnetic control of valley pseudospin in monolayer $\mathrm{WS}e_{2}$,” Nat. Phys. 11, 148 (2014).
  * Zeng  _et al._ (2012) H. Zeng, J. Dai, W. Yao, D. Xiao, and X. Cui, “Valley polarization in $\mathrm{MoS}_{2}$ monolayers by optical pumping,” Nat. Nanotechnol. 7, 490 (2012).
  * Mak  _et al._ (2012) K. Mak, K. He, J. Shan, and T. Heinz, “Control of valley polarization in monolayer $\mathrm{MoS}_{2}$ by optical helicity,” Nat. Nanotechnol. 7, 494 (2012).
  * Cao  _et al._ (2012) T. Cao, G. Wang, W. Han, H. Ye, C. Zhu, J. Shi, Q. Niu, P. Tan, E. Wang, B. Liu, and J. Feng, “Valley-selective circular dichroism of monolayer molybdenum disulphide,” Nat. Commun. 3, 887 (2012).
  * Xiao  _et al._ (2012) D. Xiao, G.B. Liu, W. Feng, X. Xu, and W. Yao, “Coupled spin and valley physics in monolayers of $\mathrm{MoS}_{2}$ and other group-vi dichalcogenides,” Phys. Rev. Lett. 108, 196802 (2012).
  * Wu  _et al._ (2013) S. Wu, J. Ross, G.B. Liu, G. Aivazian, A. Jones, Z. Fei, W. Zhu, D. Xiao, W. Yao, D. Cobden, and X. Xu, “Electrical tuning of valley magnetic moment through symmetry control in bilayer $\mathrm{MoS}_{2}$,” Nature Phys 9, 149 (2013).
  * Li  _et al._ (2014) Y. Li, J. Ludwig, T. Low, A. Chernikov, X. Cui, G. Arefe, Y. Kim, A. van der Zande, A. Rigosi, H. M. Hill, S. H. Kim, J. Hone, Z. Li, D. Smirnov, and T. F. Heinz, “Valley splitting and polarization by the zeeman effect in monolayer $\mathrm{MoSe}_{2}$,” Phys. Rev. Lett. 113, 266804 (2014).
  * MacNeill  _et al._ (2015) D. MacNeill, C. Heikes, K. Mak, Z. Anderson, A. Kormányos, V. Zólyomi, J. Park, and D. Ralph, “Breaking of valley degeneracy by magnetic field in monolayer $\mathrm{MoSe}_{2}$,” Phys. Rev. Lett. 114, 037401 (2015).
  * Srivastava  _et al._ (2015) A. Srivastava, M. Sidler, A. Allain, D. Lembke, A. Kis, and A. Imamoğlu, “Valley zeeman effect in elementary optical excitations of monolayer $\mathrm{WSe}_{2}$,” Nat. Phys. 11, 141 (2015).
  * Sallen  _et al._ (2012) G. Sallen, L. Bouet, X. Marie, G. Wang, C. R. Zhu, W. P. Han, Y. Lu, P. H. Tan, T. Amand, B. L. Liu, and B. Urbaszek, “Robust optical emission polarization in $\mathrm{MoS}_{2}$ monolayers through selective valley excitation,” Phys. Rev. B 86, 081301 (2012).
  * Kioseoglou  _et al._ (2012) G. Kioseoglou, A. T. Hanbicki, M. Currie, A. L. Friedman, D. Gunlycke, and B. T. Jonker, “Valley polarization and intervalley scattering in monolayer $\mathrm{MoS}_{2}$,” Appl. Phys. Lett. 101, 221907 (2012).
  * Guan  _et al._ (2021) S. Guan, G. Zhang, and C. Liu, “Enhanced in-plane ferroelectricity, antiferroelectricity, and unconventional $2\mathrm{D}$ emergent fermions in quadruple-layer $\mathrm{XSbO}_{2}$ ($\mathrm{X=Li,Na}$),” Nanoscale 13, 19172 (2021).
  * Guan and Luo (2020) S. Guan and J. W. Luo, “Electrically switchable hidden spin polarization in antiferroelectric crystals,” Phys. Rev. B 102, 184104 (2020).
  * Withers  _et al._ (2015) F. Withers, O. Del Pozo-Zamudio, A. Mishchenko, A. P. Rooney, A. Gholinia, K. Watanabe, T. Taniguchi, S. J. Haigh, A. K. Geim, A. I. Tartakovskii, and K. S. Novoselov, “Light-emitting diodes by band-structure engineering in van der waals heterostructures,” Nat. Mater. 14, 301 (2015).
  * Lee  _et al._ (2014) C. H. Lee, G. Hy. Lee, A. van der Zande, W. Chen, Y. Li, M. Han, X. Cui, G. Arefe, C. Nuckolls, T. Heinz, J. Guo, J. Hone, and P. Kim, “Atomically thin p–n junctions with van der waals heterointerfaces,” Nat. Nanotechnol. 9, 676 (2014).
  * Xu  _et al._ (2018) L. Xu, M. Yang, L. Shen, J. Zhou, T. Zhu, and Y. Feng, “Large valley splitting in monolayer $\mathrm{WS}_{2}$ by proximity coupling to an insulating antiferromagnetic substrate,” Phys. Rev. B 97, 041405 (2018).
  * Norden  _et al._ (2019) T. Norden, C. Zhao, P. Zhang, R. Sabirianov, A. Petrou, and H. Zeng, “Giant valley splitting in monolayer $\mathrm{WS}_{2}$ by magnetic proximity effect,” Nat. Commun. 10, 4163 (2019).
  * Peng  _et al._ (2018) R. Peng, Y. Ma, S. Zhang, B. Huang, and Y. Dai, “Valley polarization in janus single-layer $\mathrm{MoSSe}$ via magnetic doping,” J. Phys. Chem. Lett. 9, 3612 (2018).
  * Qi  _et al._ (2015) J. Qi, X. Li, Q. Niu, and J. Feng, “Giant and tunable valley degeneracy splitting in $\mathrm{MoTe}_{2}$,” Phys. Rev. B 92, 121403 (2015).
  * Tong  _et al._ (2016) W. Y. Tong, S. J. Gong, X. Wan, and C. G. Duan, “Concepts of ferrovalley material and anomalous valley hall effect,” Nat. Commun. 7, 13612 (2016).
  * Feng  _et al._ (2021) X. Feng, X. Xu, Z. He, R. Peng, Y. Dai, B. Huang, and Y. Ma, “Valley-related multiple hall effect in monolayer $\mathrm{VSi}_{2}\mathrm{P}_{4}$,” Phys. Rev. B 104, 075421 (2021).
  * He  _et al._ (2021) Z. He, R. Peng, X. Feng, X. Xu, Y. Dai, B. Huang, and Y. Ma, “Two-dimensional valleytronic semiconductor with spontaneous spin and valley polarization in single-layer $\mathrm{Cr}_{2}\mathrm{Se}_{3}$,” Phys. Rev. B 104, 075105 (2021).
  * Cheng  _et al._ (2021) H. X. Cheng, J. Zhou, W. Ji, Y. N. Zhang, and Y. P. Feng, “Two-dimensional intrinsic ferrovalley $\mathrm{Gd}\mathrm{I}_{2}$ with large valley polarization,” Phys. Rev. B 103, 125121 (2021).
  * Cui  _et al._ (2021) Q. Cui, Y. Zhu, J. Liang, P. Cui, and H. Yang, “Spin-valley coupling in a two-dimensional $\mathrm{VSi}_{2}\mathrm{N}_{4}$ monolayer,” Phys. Rev. B 103, 085421 (2021).
  * Zhao  _et al._ (2019) P. Zhao, Y. Ma, C. Lei, H. Wang, B. Huang, and Y. Dai, “Single-layer $\mathrm{LaBr}_{2}$: Two-dimensional valleytronic semiconductor with spontaneous spin and valley polarizations,” Appl. Phys. Lett. 115, 261605 (2019).
  * Jiang  _et al._ (2021) P. Jiang, L. Kang, Y. L. Li, X. Zheng, Z. Zeng, and S. Sanvito, “Prediction of the two-dimensional janus ferrovalley material $\mathrm{LaBrI}$,” Phys. Rev. B 104, 035430 (2021).
  * Guo  _et al._ (2021) S. D. Guo, J. X. Zhu, W. Q. Mu, and B. G. Liu, “Possible way to achieve anomalous valley hall effect by piezoelectric effect in a $\mathrm{GdCl}_{2}$ monolayer,” Phys. Rev. B 104, 224428 (2021).
  * Huan  _et al._ (2021) H. Huan, Y. Xue, B. Zhao, G. Gao, H. Bao, and Z. Yang, “Strain-induced half-valley metals and topological phase transitions in $\mathrm{MBr}_{2}$ monolayers ($\mathrm{M}=\mathrm{Ru}$, $\phantom{\rule{4.0pt}{0.0pt}}\mathrm{Os})$,” Phys. Rev. B 104, 165427 (2021).
  * Zang  _et al._ (2021) Y. Zang, Y. Ma, R. Peng, H. Wang, B. Huang, and Y. Dai, “Large valley-polarized state in single-layer $\mathrm{NbX}_{2}$ ($\mathrm{X}=\mathrm{S}$, $\phantom{\rule{4.0pt}{0.0pt}}\mathrm{Se}$): Theoretical prediction,” Nano Res. 14, 934 (2021).
  * Li  _et al._ (2021) S. Li, Q. Wang, C. Zhang, P. Guo, and S. A. Yang, “Correlation-driven topological and valley states in monolayer $\mathrm{VSi}_{2}\mathrm{P}_{4}$,” Phys. Rev. B 104, 085149 (2021).
  * Hu  _et al._ (2020) H. Hu, W. Tong, Y. Shen, X. Wan, and C. G. Duan, “Concepts of the half-valley-metal and quantum anomalous valley hall effect,” npj Quantum Mater. 6, 129 (2020).
  * Zhao  _et al._ (2021) Yi. F. Zhao, Y. H. Shen, H. Hu, W. Y. Tong, and C. G. Duan, “Combined piezoelectricity and ferrovalley properties in janus monolayer $\mathrm{VClBr}$,” Phys. Rev. B 103, 115124 (2021).
  * Liu  _et al._ (2020a) X. Liu, A. Pyatakov, and W. Ren, “Magnetoelectric coupling in multiferroic bilayer $\mathrm{VS}_{2}$,” Phys. Rev. Lett. 125, 247601 (2020a).
  * Du  _et al._ (2020) W. Du, Y. Ma, R. Peng, H. Wang, B. Huang, and Y Dai, “Prediction of single-layer $\mathrm{TiVI}_{6}$ as a promising two-dimensional valleytronic semiconductor with spontaneous valley polarization,” J. Mater. Chem. C 8, 13220 (2020).
  * Liu  _et al._ (2017) J. Liu, W. J. Hou, C. Cheng, H. X. Fu, J. T. Sun, and S. Meng, “Intrinsic valley polarization of magnetic $\mathrm{VSe}_{2}$ monolayers,” Journal of Physics: Condensed Matter 29, 255501 (2017).
  * Blöchl (1994) P. E. Blöchl, “Projector augmented-wave method,” Phys. Rev. B 50, 17953 (1994).
  * Kresse and Furthmüller (1996) G. Kresse and J. Furthmüller, “Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set,” Phys. Rev. B 54, 11169 (1996).
  * Perdew  _et al._ (1996) J. Perdew, K. Burke, and M. Ernzerhof, “Generalized gradient approximation made simple,” Phys. Rev. Lett. 77, 3865–3868 (1996).
  * Pizzi  _et al._ (2020) G. Pizzi, V. Vitale, R. Arita, S. Blügel, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, J. Ibañez-Azpiroz, H. Lee, J.-M. Lihm, D. Marchand, A. Marrazzo, Y. Mokrousov, J. I. Mustafa, Y. Nohara, Y. Nomura, and L. Paulatto et al., “Wannier90 as a community code: new features and applications,” J. Physics.: Condens. Matter 32, 165902 (2020).
  * Wu  _et al._ (2018) Q. Wu, S. Zhang, H. F. Song, M. Troyer, and A. Soluyanov, “WannierTools: An open-source software package for novel topological materials,” Comput. Phys. Commun. 224, 405–416 (2018).
  * Hong  _et al._ (2020) Y. L. Hong, Z. Liu, L. Wang, T. Zhou, W. Ma, C. Xu, S. Feng, L. Chen, M. L. Chen, D. M. Sun, X. Q. Chen, H. M. Cheng, and W. Ren, “Chemical vapor deposition of layered two-dimensional $\mathrm{MoSi}_{2}\mathrm{N}_{4}$ materials,” Science 369, 670 (2020).
  * Zhang  _et al._ (2019) C. Zhang, Y. Nie, S. Sanvito, and A. Du, “First-principles prediction of a room-temperature ferromagnetic janus $\mathrm{VSSe}$ monolayer with piezoelectricity, ferroelasticity, and large valley polarization,” Nano Lett. 19, 1366 (2019).
  * Wu  _et al._ (2021) Y. Wu, W. Sun, S. Liu, B. Wang, C. Liu, H. Yin, and Z. Cheng, “$\mathrm{Ni(NCS)}_{2}$ monolayer: a robust bipolar magnetic semiconductor,” Nanoscale 13, 16564 (2021).
  * Goodenough (1955) John B. Goodenough, “Theory of the role of covalence in the perovskite-type manganites $[\mathrm{La},\mathrm{M}(\mathrm{II})]\mathrm{MnO}_{3}$,” Phys. Rev. 100, 564 (1955).
  * Anderson (1959) P. W. Anderson, “New approach to the theory of superexchange interactions,” Phys. Rev. 115, 2 (1959).
  * Kanamori (1960) J. Kanamori, “Crystal distortion in magnetic compounds,” J. Appl. Phys. 31, S14 (1960).
  * Huang  _et al._ (2017) B. Huang, G. Clark, E. Navarro-Moratall, D. Klein, R. Cheng, K. Seyler, D. Zhong, E. Schmidgall, M. McGuire, D. Cobden, P. Jarillo-Herrero W. Yao, D. Xiao, and X. Xu, “Layer-dependent ferromagnetism in a van der waals crystal down to the monolayer limit,” Nature 546, 270 (2017).
  * Gong  _et al._ (2017) C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y. Wang, Z. Q. Qiu, R. J. Cava, S. Louie, J. Xia, and X. Zhang, “Discovery of intrinsic ferromagnetism in two-dimensional van der waals crystals,” Nature 546, 265 (2017).
  * Li  _et al._ (2020) S. Li, W. Wu, X. Feng, S. Guan, W. Feng, Y. Yao, and S. A. Yang, “Valley-dependent properties of monolayer $\mathrm{MoSi}_{2}\mathrm{N}_{4}$, $\mathrm{WSi}_{2}\mathrm{N}_{4}$, and $\mathrm{MoSi}_{2}\mathrm{As}_{4}$,” Phys. Rev. B 102, 235435 (2020).
  * Guan  _et al._ (2017) S. Guan, Z. M. Yu, Y. Liu, G. B. Liu, L. Dong, Y. Lu, and Y. Yao, “Artificial gravity field, astrophysical analogues, and topological phase transitions in strained topological semimetals,” npj Quantum Mater. 2, 23 (2017).
  * Liu  _et al._ (2020b) C. Liu, B. Fu, H. Yin, G. Zhang, and C. Dong, “Strain-tunable magnetism and nodal loops in monolayer $\mathrm{MnB}$,” Appl. Phys. Lett. 117, 103101 (2020b).
  * Liu  _et al._ (2019) C. Liu, S. Guan, H. Yin, W. Wan, Y. Wang, and Y. Zhang, “$\gamma-\mathrm{GeSe}$: A two-dimensional ferroelectric material with doping-induced ferromagnetism,” Appl. Phys. Lett. 115, 252904 (2019).
  * Liu  _et al._ (2021a) P. Liu, G. Zhang, Y. Yan, G. Jia, C. Liu, B. Wang, and H. Yin, “Strain-tunable phase transition and doping-induced magnetism in iodinene,” Appl. Phys. Lett. 119, 102403 (2021a).
  * Liu  _et al._ (2021b) C. Liu, B. Wang, G. Jia, P. Liu, H. Yin, S. Guan, and Z. Cheng, “Tunable magnetism in ferroelectric $\alpha-\mathrm{In2Se3}$ by hole-doping,” Appl. Phys. Lett. 118, 072902 (2021b).
  * Liu  _et al._ (2022) S. Liu, H. Yin, D. J. Singh, and P. F. Liu, “$\mathrm{Ta}_{4}\mathrm{SiTe}_{4}$: A possible one-dimensional topological insulator,” Phys. Rev. B 105, 195419 (2022).
  * Xiao  _et al._ (2010) D. Xiao, M. C. Chang, and Q. Niu, “Berry phase effects on electronic properties,” Rev. Mod. Phys. 82, 1959 (2010).


# Deep Gradient Projection Networks for Pan-sharpening

Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, Chunxia Zhang
School of Mathematics and Statistics, Xi’an Jiaotong University, Xi’an 710049,
China
<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Pan-sharpening is an important technique for remote sensing imaging systems to
obtain high resolution multispectral images. Recently, deep learning has
become the most popular tool for pan-sharpening. This paper develops a model-
based deep pan-sharpening approach. Specifically, two optimization problems
regularized by the deep prior are formulated, and they are separately
responsible for the generative models for panchromatic images and low
resolution multispectral images. Then, the two problems are solved by a
gradient projection algorithm, and the iterative steps are generalized into
two network blocks. By alternatively stacking the two blocks, a novel network,
called gradient projection based pan-sharpening neural network, is
constructed. The experimental results on different kinds of satellite datasets
demonstrate that the new network outperforms state-of-the-art methods both
visually and quantitatively. The codes are available at
https://github.com/xsxjtu/GPPNN.

##  1 Introduction

Figure 1: (a) The observation models for LRMS and PAN images. (b) Two
formulated optimization problems and iterative steps of the gradient
projection algorithm. (c) The main blocks in our proposed GPPNN.

Multispectral images store multiple images corresponding to each band (or say,
channel) in an optical spectrum, and they are widely utilized in literature of
remote sensing. With the limitation of imaging devices, satellites however
often measure the low spatial resolution multispectral (LRMS) images [4, 21,
29]. Compared with the multispectral image, the panchromatic (PAN) image is
characterized by the high spatial resolution but only one band. Lots of
satellites carry both multispectral and panchromatic sensors to simultaneously
measure the complementary images, such as Landsat8, GaoFen2 and QuickBird. To
obtain the high resolution multispectral (HRMS) image, a promising way is to
fuse the complementary information of the LRMS image and the PAN image. This
technique is called as pan-sharpening [4, 21].

Pan-sharpening can be cast as a typical image fusion on super-resolution
problems. The past decades witnessed the development of pan-sharpening. The
classic algorithms including component substitution (CS) [7, 12],
multiresolution analysis (MRA) [1, 23] and other techniques. In the era of
deep learning, convolutional neural networks have emerged as a significant
tool for pan-sharpening [19]. One of the seminal work is the pan-sharpening
neural network (PNN) proposed by Masi _et al_. [20]. Borrowing the idea of the
first super-resolution network [2], PNN is fed with the concatenation of a PAN
image and an upsampled LRMS image to regress the HRMS image.

In fact, there are only three convolutional units in PNN, so it is a
relatively shallow network. Recently, numerous models have been proposed to
improve the PNN. Owing to the triumphs of residual networks [8], several
papers utilize the shortcut or residual convolutional units to build deep
networks, including MIPSM [16], DRPNN [34] and PanNet [38]. They generally
contain 10 or more convolutional units. Besides these networks, to make the
best of advantages of neural networks, some researchers build deeper networks.
For example, Wang _et al_. employ the densely connected convolutional unit
[10] to design a 44-layer network [31] for pan-sharpening.

It is well-known that deepening the layers of networks does not necessarily
improve the performance, since it is difficult to train deeper networks and
redundant parameters make them easily over-fit. Very recently, the remote
sensing community begins to rethink how to make the full use of PAN images’
information [9, 24]. It is worthy noting that most the pan-sharpening networks
regard the PAN image as a channel of the input. This manner ignores different
characteristics between PAN and LRMS images. A growing number of researchers
attempt to propose the two-branch networks [18, 40]. In the first stage, the
two branches separately extract the features for PAN and LRMS images. In the
second stage, the features are fused to reconstruct the HRMS image.

Although convolutional neural networks exhibit promising performance in pan-
sharpening, they require a large amount of training samples [22, 33], and they
do not account for the observation progress of PAN and LRMS images, i.e.,
lacking the interpretability. Therefore, there still leaves the room for
improvement. The research on model-based deep learning is the trend in image
processing field to close the gap between classic models and neural networks,
and it is found that model-based deep networks usually outperform the
intuitively designed networks [22, 33]. Xie _et al_. present a multispectral
and hyperspectral (HS) image fusion network (MHNet) for the hyperspectral pan-
sharpening task [35]. There is no doubt that MHNet can be naturally adapted to
pan-sharpening [36]. Nonetheless, MHNet is designed to describe the low-rank
property for hyperspectral images, and our experiments show that MHNet may
perform badly in the pan-sharpening scenario.

In this paper, we develop a novel model-based deep network for pan-sharpening.
Our contributions are summarized as follows:

Firstly, this paper considers the generative models for PAN and LRMS images.
That is, as shown in Fig. 1(a), PAN images are the linear combination of the
bands in HRMS images, and LRMS images are generated by blurring and
downsampling HRMS images. Combining the observation models and the deep prior,
we propose two optimization problems, and they can be effectively solved by
the gradient projection method as illustrated in Fig. 1(b).

Secondly, inspired by the idea of algorithm unrolling techniques, the
iterative steps are generalized as two neural blocks separately justifying the
generative models for PAN and LRMS images. The computational flows in the
proposed neural blocks are interpretable. As show in Fig. 1(c), for the MS
Block, given a current estimation of the HRMS image, it generates
corresponding LRMS image and computes the residual between the generated LRMS
image and the real one. This residual then is upsampled and is added into the
current estimation to reconstruct the next HRMS image. The PAN block can be
interpreted similarly. We build a new network by alternatively stacking the
two blocks. In what follows, it calls the gradient projection based pan-
sharpening neural network (GPPNN). To the best of our knowledge, it is the
first model-driven deep network for pan-sharpening.

Thirdly, the proposed GPPNN is compared with the 13 state-of-the-art (SOTA)
and classic pan-sharpening methods. The extensive experiments conducted on
three popular satellites (i.e., Landsat8, QuickBird, GF2) demonstrate that our
networks outperform other counterparts both quantitatively and visually.

##  2 Related work

###  2.1 Classic pan-sharpening methods

The classic pan-sharpening methods mainly consists of CS based algorithms, MRA
based algorithms and other algorithms. CS methods assume that the spatial and
spectral information of a multispectral image can be decomposed. Therefore, an
HRMS image is reconstructed by combining the spatial information of a PAN
image and the spectral information of an LRMS image. In the past decades,
researchers have designed various decomposition algorithms. For example,
intensity-hue-saturation (IHS) fusion [7] employs the IHS transformation,
Brovey method [5] uses a multiplicative injection scheme, and Gram-Schmidt
(GS) method [12] exploits the Gram-Schmidt orthogonalization procedure. The
main drawback of CS methods is that the image contains artifacts if the
spectral and spatial information is not appropriately decomposed. The MRA
methods apply the multi-scale transformation to PAN images to extract spatial
details which then are injected into the upsampled LRMS images. Typical
algorithms include high-pass filter (HPF) fusion [26], and Indusion method
[11], smoothing filter-based intensity modulation (SFIM) [15]. The performance
of the MRA method strongly depends on the multi-scale transformation.

###  2.2 Deep learning based methods

Recently, convolutional neural networks have been one of the most effective
tools for remote sensing. Given a parameterized network, it is fed with an
LRMS image and a PAN image to regress an HRMS image, and its parameters (or
say, weights) are learned from data in the end-to-end fashion. The first
attempt is the PNN with three convolutional units [20]. Recently, thanks to
the rapid development of computer vision [8, 10], it is able to train very
deep networks. Researchers propose the deep pan-sharpening networks with
dozens of layers and the performance has been greatly improved [27, 31, 38,
40]. At the same time, researchers also explore the two-branch networks to
separately extract the features from MS and PAN images [18, 40]. Recently, one
of the research trends of the pan-sharpening community is to combine the
classic methods with deep neural networks to improve the interpretability of
the deep learning based methods. For example, inspired by the idea of MRA
algorithms, MIPSM [16] designs a spatial detail extraction network for the PAN
images and injects the details into the LRMS images. Liu _et al_. propose an
adaptive weight network for integrating the advantages of different classic
methods [14]. It overcomes the shortcomings of the CS and MRA algorithms, and
outperforms some SOTA deep learning based methods.

###  2.3 Model-driven deep networks

Most of the deep neural networks are designed intuitively. Recently, a growing
number of researchers focus on model-based neural networks for image
processing tasks [22, 33]. The basic idea of model-driven deep learning is to
formulate an observation model or optimization problem by integrating the
prior knowledge for a specific task and to translate each iteration of the
algorithm step into a layer of deep neural networks [22, 33]. Passing through
the stacked layers corresponds to execute the algorithm with a certain number
of times. Model-based deep learning builds the bridge between classic models
and deep neural networks. This idea has been successfully applied in various
tasks, including sparse coding [6], compressive sensing [39], image deblurring
[13], image dehazing [37] and image deraining [32]. It is worth mentioning the
MHNet, a model-driven network for the hyperspectral pan-sharpening task [35]
to super-resolve HS images with the guidance of MS images. It can be naturally
adapted to pan-sharpening, but MHNet mainly focuses on the low-rank property
for HS images, i.e., its rank $r_{\rm HS}$ is far lower than the number of
bands $B_{\rm HS}$. In practice, there are dozens or hundreds of bands in an
HS image, while there are no more than 10 bands in an MS image. So, the low-
rank property is not evident for MS images, and MHNet may break down in pan-
sharpening task.

##  3 GPPNN

In this section, we develop a model-driven network for pan-sharpening. For
convenience, we summarize the notations in this paper before presentation of
the GPPNN. $\bm{L}\in R^{mn\times B}$ is an LRMS image with a height of $m$, a
width of $n$ and the number of bands of $B$. $\bm{H}\in R^{MN\times B}$ is an
HRMS image with a height of $M$, a width of $N$ and the number of bands of
$B$. $\bm{P}\in R^{MN\times b}$ is a PAN image whose spatial resolution is the
same with that of $\bm{H}$, but there is only one band (i.e., $b=1$).
$r=M/m=N/n$ is the spatial resolution ratio. With abuse of notations, we use
their tensor versions in the context of deep learning (namely, $\mathcal{L}\in
R^{m\times n\times B},\mathcal{H}\in R^{M\times N\times B},\mathcal{P}\in
R^{M\times N\times b}$). Notation $\mathtt{Conv}(\cdot;c^{\rm in},c^{\rm
out})$ is the convolutional operator whose input and output are with $c^{\rm
in}$ and $c^{\rm out}$ channels, respectively. In what follows, the function
$\mathtt{Conv}(\cdot;c^{\rm in},c^{\rm mid},c^{\rm out})$ denotes the cascaded
convolutional operator, that is,

$\displaystyle\mathtt{Conv}\left(\cdot;c^{\rm in},c^{\rm mid},c^{\rm
out}\right)$ (1) $\displaystyle=$
$\displaystyle\mathtt{Conv}\left(\mathtt{ReLU}\left(\mathtt{Conv}\left(\cdot;c^{\rm
in},c^{\rm mid}\right)\right);c^{\rm mid},c^{\rm out}\right).$

###  3.1 Model formulation

Our network starts with the observation model for the LRMS, HRMS and PAN
images. It is assumed that an LRMS image is obtained by downsampling and
blurring an HRMS image, while a PAN image is the result of spectral response
for an HRMS image. In formula, we have $\bm{L}=\bm{DKH},\bm{P}=\bm{HS},$ where
$\bm{D}\in R^{mn\times MN}$ denotes a downsampling matrix and $\bm{K}$ is the
(low-passing) circular convolution matrix, and $\bm{S}\in R^{B\times b}$ is
the so-called spectral response function. It is well-known that inferring the
HRMS image is an ill-posed problem. Hence, it often formulates the following
penalized optimization,

$\min_{\bm{H}}f(\bm{L},\bm{H})+g(\bm{P},\bm{H})+\lambda h(\bm{H}),$ (2)

where $h(\cdot)$ is the prior term, and
$f(\bm{L},\bm{H})=\|\bm{L}-\bm{DKH}\|_{2}^{2}/2$ and
$g(\bm{P},\bm{H})=\|\bm{P}-\bm{HS}\|_{2}^{2}/2$ are data fidelity terms which
are responsible to LRMS and PAN images, respectively. In the classic methods,
$h(\cdot)$ is usually designed as a hand-craft function, such as the total
variation or nuclear norm [17]. However, in the era of deep learning, it is
suggested to set $h(\cdot)$ as a deep prior [28, 41]. In other words, it is
better to set an implicit prior captured by the neural network
parametrization. Additionally, the deep prior is learned from data and can
adapt to different tasks and observation models. To make the best of deep
prior, instead of the above issue, we consider an LRMS-aware problem and a
PAN-aware problem:

$\displaystyle\min_{\bm{H}}$
$\displaystyle\frac{1}{2}\left\|\bm{L}-\bm{DKH}\right\|_{2}^{2}+\lambda
h_{l}(\bm{H}),$ (3a) $\displaystyle\min_{\bm{H}}$
$\displaystyle\frac{1}{2}\left\|\bm{P}-\bm{HS}\right\|_{2}^{2}+\lambda
h_{p}(\bm{H}).$ (3b)

Here, $h_{l}(\cdot)$ and $h_{p}(\cdot)$ are two deep priors accounting for the
observations of LRMS and PAN images, respectively. The ablation experiment in
section 4.4 verifies that Eq. (3) achieves better results than Eq. (2). In the
next, we describe how to solve the two problems. Moreover, the solutions are
generalized into an LRMS-aware block (MS Block) and a PAN-aware block (PAN
block) that can be embedded into neural networks.

###  3.2 MS Block

We employ the gradient projection method [25] to solve Eq. (3a) and the
updating rule is

$\bm{H}^{(t)}=\mathrm{prox}_{h_{l}}\left(\bm{H}^{(t-1)}-\rho\nabla
f(\bm{H}^{(t-1)})\right),$ (4)

where $\rho$ is the step size, $\mathrm{prox}_{h_{l}}(\cdot)$ is a proximal
operator corresponding to penalty ${h_{l}}(\cdot)$ and $\nabla
f(\bm{H}^{(t-1)})=-(\bm{DK})^{T}(\bm{L}-\bm{DKH})$ denotes the gradient of the
data fidelity term.

Inspired by the principle of model-driven deep learning [22], we generalize
Eq. (4) as a network block. To begin with, Eq. (4) is split into four steps as
follows,

$\displaystyle\hat{\bm{L}}^{(t)}$ $\displaystyle=\bm{DK}\bm{H}^{(t-1)},$ (5a)
$\displaystyle\bm{R}_{l}^{(t)}$ $\displaystyle=\bm{L}-\hat{\bm{L}}^{(t)},$
(5b) $\displaystyle\bm{R}_{h}^{(t)}$
$\displaystyle=\rho\left(\bm{DK}\right)^{T}\bm{R}_{l}^{(t)},$ (5c)
$\displaystyle\bm{H}^{(t)}$
$\displaystyle=\mathrm{prox}_{h_{l}}\left(\bm{H}^{(t-1)}+\bm{R}_{h}^{(t)}\right).$
(5d)

Then, each step is translated with deep learning terminologies. For
convenience, we use the tensor versions to represent the variables in the
context of deep learning. In Eq. (5a), given a current HRMS image
$\mathcal{H}^{(t-1)}$, it generates an LRMS image $\hat{\mathcal{L}}^{(t)}$ by
applying a low-passing filter and downsampling. In neural networks, this step
is implemented by

$\hat{\mathcal{L}}^{(t)}=\mathtt{Conv}\left(\mathcal{H}^{(t-1)};B,C,B\right)\downarrow,$
(6)

where downsampling is conducted with a bicubic interpolation $\downarrow$ and
the filter $\bm{K}$ is replaced by a cascaded convolutional operator
$\mathtt{Conv}(\cdot;B,C,B)$ to obtain more expressive features. $C$ is the
number of channels for the feature maps, and we set it to 64 in this paper.
$B$ is the number of channels for MS images, and it depends on the input data.

Afterwards, Eq. (5b) computes residuals between the real LRMS image
$\mathcal{L}$ and the generated LRMS image $\hat{\mathcal{L}}^{(t)}$, and the
translation is trivial as shown in following equation,

$\mathcal{R}_{l}^{(t)}=\mathcal{L}-\hat{\mathcal{L}}^{(t)}.$ (7)

In the next, Eq. (5c) obtains the high-resolution residuals. Analogous to Eqs.
(5a) and (6), this step is rewritten as

$\mathcal{R}_{h}^{(t)}=\rho\mathtt{Conv}\left(\mathcal{R}_{l}^{(t)};B,C,B\right)\uparrow.$
(8)

Remark that the filters in Eqs. (5a) and (5c) transpose to each other, but we
do not force the convolutional kernels in Eqs. (6) and (8) to satisfy this
requirement for flexibility. The ablation experiment in section 4.4 shows that
it slightly improves GPPNN’s performance. At last, Eq. (5d) outputs the HRMS
image by taking the residual into account with a proximal operator. As
illustrated before, proximal operators regarding the deep prior are modeled by
the deep networks [28, 41]. In this manner, the deep prior can be learned
implicitly from data. So, we have

$\mathcal{H}^{(t)}=\mathtt{Conv}\left(\mathcal{H}^{(t-1)}+\mathcal{R}^{(t)}_{h};B,C,B\right).$
(9)

In what follows, Eqs. (6), (7), (8) and (9) are named as an MS Block. For
better understanding, the computational flow for an MS Block is displayed in
Fig. 1(c).

Figure 2: The structure of GPPNN.

###  3.3 PAN block

In this subsection, we consider the observation model for PAN (i.e., Eq.
(3b)). With the gradient projection method, the updating rule is

$\bm{H}^{(t)}=\mathrm{prox}_{h_{p}}\left(\bm{H}^{(t-1)}-\rho\nabla
g(\bm{H}^{(t-1)})\right),$ (10)

where $\nabla g(\bm{H}^{(t-1)})=-(\bm{P}-\bm{HS})\bm{S}^{T}$. With the similar
techniques, it is able to translate Eq. (10) into a block of neural networks.
At first, Eq. (10) is split into four steps as follows,

$\displaystyle\hat{\bm{P}}^{(t)}$ $\displaystyle=\bm{H}^{(t-1)}\bm{S},$ (11a)
$\displaystyle\bm{R}_{p}^{(t)}$ $\displaystyle=\bm{P}-\hat{\bm{P}}^{(t)},$
(11b) $\displaystyle\bm{R}_{h}^{(t)}$
$\displaystyle=\rho\bm{R}_{p}^{(t)}\bm{S}^{T},$ (11c)
$\displaystyle\bm{H}^{(t)}$
$\displaystyle=\mathrm{prox}_{h_{p}}\left(\bm{H}^{(t-1)}+\bm{R}_{h}^{(t)}\right).$
(11d)

In the context of deep learning, as shown in Fig. 1(c), these steps are
rewritten as,

$\displaystyle\hat{\mathcal{P}}^{(t)}$
$\displaystyle=\mathtt{Conv}\left(\mathcal{H}^{(t-1)};B,C,b\right),$ (12a)
$\displaystyle\mathcal{R}_{p}^{(t)}$
$\displaystyle=\mathcal{P}-\hat{\mathcal{P}}^{(t)},$ (12b)
$\displaystyle\mathcal{R}_{h}^{(t)}$
$\displaystyle=\rho\mathtt{Conv}\left(\mathcal{R}_{p}^{(t)};b,C,B\right),$
(12c) $\displaystyle\mathcal{H}^{(t)}$
$\displaystyle=\mathtt{Conv}\left(\mathcal{H}^{(t-1)}+\mathcal{R}^{(t)}_{h};B,C,B\right).$
(12d)

Here, $b=1$ is the number of channel for PAN images. Remark that the
underlying assumption of Eq. (3b) is that the PAN image is a linear
combination of the HRMS image. $\bm{S}$/$\bm{S}^{T}$ is regarded as a band
reduction/expansion operator. With this assumption, convolutional units in
Eqs. (12a) and (12c) should be with the kernel size of 1.

###  3.4 GPPNN

Now, with the MS Block and the PAN block, we are ready to construct the
gradient projection based pan-sharpening neural network (GPPNN). The structure
of our GPPNN is shown in Fig. 2. The network starts with an input layer, and
it requires an initial value of the HRMS image. We initialize
$\mathcal{H}^{(0)}\in R^{M\times N\times B}$ by applying the bicubic
interpolation to the input LRMS image $\mathcal{L}\in R^{m\times n\times B}$.
The network is followed by a backbone subnetwork. There are $K$ layers, each
of which consists of an MS Block and a PAN block. In formula, there are

$\mathcal{H}^{(t+0.5)}=\mathtt{MS\\_Block}\left(\mathcal{H}^{(t)},\mathcal{L}\right)$
(13)

and

$\mathcal{H}^{(t+1)}=\mathtt{PAN\\_Block}\left(\mathcal{H}^{(t+0.5)},\mathcal{P}\right).$
(14)

The output of the last layer, denoted by $\hat{\mathcal{H}}\in R^{M\times
N\times B}$, is the final reconstructed HRMS.

###  3.5 Training details

Our GPPNN is supervised by the $\ell_{1}$ loss between $\hat{\mathcal{H}}$ and
the ground truth $\mathcal{H}$, $\|\hat{\mathcal{H}}-\mathcal{H}\|_{1}.$ The
paired training samples are unavailable in practice. When we construct the
training set, the Wald protocol [30] is employed to generate the paired
samples. For example, given the multispectral image $\mathcal{H}\in R^{M\times
N\times B}$ and the PAN image $\mathcal{P}\in R^{rM\times rN\times b}$, both
of them are downsampled with ratio $r$, and the downsampled versions are
denoted by $\mathcal{L}\in R^{M/r\times N/r\times B}$ and
$\tilde{\mathcal{P}}\in R^{M\times N\times b}$. In the training set,
$\mathcal{L}$ and $\tilde{\mathcal{P}}$ are regarded as the inputs, while
$\mathcal{H}$ is the ground-truth.

GPPNN is implemented with Pytorch framework. They are optimized by Adam over
100 epochs with a learning rate of $5\times 10^{-4}$ and a batch size of 16.
In our experiments, $k_{\rm LR}=3$ and $k_{\rm PAN}=1$. In section 4.2, we
report the performance of GPPNN with different $C$ and $K$. As a balance, $C$
and $K$ are set to 64 and 8, respectively.

(a) LRMS

(b) PAN

(c) GT(PSNR)

(d) BDSD(17.23)

(e) GS(14.21)

(f) MIPSM(19.80)

(g) DRPNN(21.45)

(h) MSDCNN(21.94)

(i) RSIFNN(17.14)

(j) PANNET(20.09)

(k) MHNet(19.58)

(l) GPPNN(24.40)

Figure 3: Visual inspection on Landsat8 dataset. The caption of each subimage
displays the corresponding PSNR value. Table 1: The information of datasets.
$B$ is the number of bands for multispectral images.

| Landsat8 | GaoFen2 | QuickBird
---|---|---|---
$B$ | 10 | 4 | 4
Resolution-MS | 256 | 256 | 256
Resolution-PAN | 512 | 1024 | 1024
# Train/Val/Test | 350/50/100 | 350/50/100 | 474/103/100

Table 2: The PSNR values on validation datasets for GPPNN with different $K$
and $C$. The best value is highlighted by the bold.

Satellites |  $K$ layers |  $C$ Filters
---|---|---
2 | 4 | 6 | 8 | 10 | 12 | 14 | 8 | 16 | 32 | 64 | 128
Landsat8 | 39.0648 | 39.5878 | 39.9876 | 40.0368 | 40.1336 | 39.9531 | 40.0509 | 36.6455 | 39.6156 | 39.6702 | 40.0368 | 39.0841 [t]
QuickBird | 30.4994 | 30.4392 | 30.6370 | 30.5636 | 30.4803 | 30.4773 | 30.5560 | 30.2962 | 30.4681 | 30.4592 | 30.5636 | 30.5979
GaoFen2 | 36.7583 | 36.9740 | 36.2181 | 37.5606 | 37.0589 | 36.7835 | 36.6840 | 35.8116 | 36.9061 | 36.2810 | 37.5606 | 36.5873 [b]

Table 3: The four metrics on test datasets. The best and the second best
values are highlighted by the bold and underline, respectively. The up or down
arrow indicates higher or lower metric corresponds to better images.

| Landsat8 | QuickBird | GaoFen2
---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | SAM$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | SAM$\downarrow$ | ERGAS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | SAM$\downarrow$ | ERGAS$\downarrow$
BDSD | 33.8065 | 0.9128 | 0.0255 | 1.9128 | 23.5540 | 0.7156 | 0.0765 | 4.8874 | 30.2114 | 0.8732 | 0.0126 | 2.3963 [t]
Brovey | 32.4030 | 0.8533 | 0.0206 | 1.9806 | 25.2744 | 0.7370 | 0.0640 | 4.2085 | 31.5901 | 0.9033 | 0.0110 | 2.2088
GS | 32.0163 | 0.8687 | 0.0304 | 2.2119 | 26.0305 | 0.6829 | 0.0586 | 3.9498 | 30.4357 | 0.8836 | 0.0101 | 2.3075
HPF | 32.6691 | 0.8712 | 0.0250 | 2.0669 | 25.9977 | 0.7378 | 0.0588 | 3.9452 | 30.4812 | 0.8848 | 0.0113 | 2.3311
IHS | 32.8772 | 0.8615 | 0.0245 | 2.3128 | 24.3826 | 0.6742 | 0.0647 | 4.6208 | 30.4754 | 0.8639 | 0.0108 | 2.3546
Indusion | 30.8476 | 0.8168 | 0.0359 | 2.4216 | 25.7623 | 0.6377 | 0.0674 | 4.2514 | 30.5359 | 0.8849 | 0.0113 | 2.3457
SFIM | 32.7207 | 0.8714 | 0.0248 | 2.0775 | 24.0351 | 0.6409 | 0.0739 | 4.8282 | 30.4021 | 0.8501 | 0.0129 | 2.3688
MIPSM | 35.4891 | 0.9389 | 0.0209 | 1.5769 | 27.7323 | 0.8411 | 0.0522 | 3.1550 | 32.1761 | 0.9392 | 0.0104 | 1.8830
DRPNN | 37.3639 | 0.9613 | 0.0173 | 1.3303 | 31.0415 | 0.8993 | 0.0378 | 2.2250 | 35.1182 | 0.9663 | 0.0098 | 1.3078
MSDCNN | 36.2536 | 0.9581 | 0.0176 | 1.4160 | 30.1245 | 0.8728 | 0.0434 | 2.5649 | 33.6715 | 0.9685 | 0.0090 | 1.4720
RSIFNN | 37.0782 | 0.9547 | 0.0172 | 1.3273 | 30.5769 | 0.8898 | 0.0405 | 2.3530 | 33.0588 | 0.9588 | 0.0112 | 1.5658
PANNET | 38.0910 | 0.9647 | 0.0152 | 1.3021 | 30.9631 | 0.8988 | 0.0368 | 2.2648 | 34.5774 | 0.9635 | 0.0089 | 1.4750
MHNet | 37.0049 | 0.9566 | 0.0189 | 1.3509 | 31.1557 | 0.8947 | 0.0368 | 2.1931 | 33.8930 | 0.9291 | 0.0176 | 1.3697
GPPNN | 38.9939 | 0.9727 | 0.0138 | 1.2483 | 31.4973 | 0.9075 | 0.0351 | 2.1058 | 35.9680 | 0.9725 | 0.0084 |  1.2798  [b]

##  4 Experiments

A series of experiments are carried out to evaluate the performance of GPPNN.
SOTA deep learning based methods are selected for comparison, namely, MIPSM
[16], DRPNN [34], MSDCNN [40], RSIFNN [27], PanNet [38], and MHNet [35]. Our
method is also compared with seven classic methods, including BDSD [3], Brovey
[5], GS [12], HPF [26], IHS fusion [7], Indusion [11], SFIM [15]. The
experiments are conducted on a computer with an Intel i7-9700K CPU at 3.60GHz
and an NVIDIA GeForce RTX 2080ti GPU.

(a) LRMS

(b) PAN

(c) GT(PSNR)

(d) BDSD(16.23)

(e) GS(15.80)

(f) MIPSM(16.84)

(g) DRPNN(18.85)

(h) MSDCNN(18.15)

(i) RSIFNN(17.33)

(j) PANNET(18.22)

(k) MHNet(19.50)

(l) GPPNN(21.13)

Figure 4: Visual inspection on QuickBird dataset. The caption of each subimage
displays the corresponding PSNR value.

###  4.1 Datasets and metrics

Remote sensing images acquired by three satellites are used in our
experiments, including Landsat8, QuickBird and GaoFen2, the basic information
of which is listed in Table 1. For each satellite, we have hundreds of image
pairs, and they are divided into three parts for training, validation and
test. Note that we determine $K$ and $C$ on the validation dataset. In the
training set, the multispectral images are cropped into patches with the size
of $32\times 32$, and the corresponding PAN patches are with the size of
$64\times 64$ (for Landsat8) or $128\times 128$ (for GaoFen2 and QuickBird).
For numerical stability, each patch is normalized by dividing the maximum
value to make the pixels range from 0 to 1.

Four popular metrics are used to evaluate the algorithms’ performances,
including peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and
erreur relative globale adimensionnelle de synthese (ERGAS) and spectral angle
mapper (SAM). The first three metrics measure the spatial distortion, and the
last one measures the spectral distortion. An image is better if its PSNR and
SSIM are higher, and ERGAS and SAM are lower.

###  4.2 The effect of depth and width

The network’s depth $K$ and width $C$ play significant roles. Table 2 lists
the PSNR values on validation datasets for GPPNN with different $K$ and $C$.
At first, $C$ is fixed to 64, and $K$ is set to 2, 4, $\cdots$, 14. It is
shown that more layers do not necessarily increase the PSNR value, and $K=8$
strikes the balance between performance and the number of weights. The reason
may be that it is not easy to train a GPPNN with more layers. Then, we fix $K$
to 8 and set $C$ to 8, 16, 32, 64 and 128. The similar conclusion can be
drawn, and the best choice for $C$ is 64. In summary, our GPPNN is configured
with $K=8$ layers and $C=64$ filters in the next experiments.

###  4.3 Comparison with SOTA methods

The evaluation metrics on three datasets are reported in Table 3. It is found
that GPPNN outperforms other methods regarding all metrics on three
satellites. Figs. 3, 4 and 5 show the RGB bands of the three satellites for
some representative methods. Our GPPNN is the closest to the ground truth.
From the amplified local regions in Fig. 3, it found that BDSD, GS, MIPSM,
DRPNN, PANNET suffer from spatial distortion, and GS, MSDCNN, RSIFNN and MHNet
suffer from spectral distortion. However, our GPPNN has the smallest spatial
and spectral distortions. As for Fig. 4, it is a difficult case. It is shown
that the most methods have obvious artifacts or noise, and their images are
blurring or spectrally distorted. Our GPPNN is without artifacts, noise or
spectral distortion. As shown in Fig. 5, it is observed that compared with
other methods, our GPPNN has finer-grained textures and coarser-grained
structures.

(a) LRMS

(b) PAN

(c) GT(PSNR)

(d) BDSD(21.86)

(e) GS(21.43)

(f) MIPSM(22.11)

(g) DRPNN(26.41)

(h) MSDCNN(26.90)

(i) RSIFNN(24.74)

(j) PANNET(26.72)

(k) MHNet(25.17)

(l) GPPNN(28.70)

Figure 5: Visual inspection on GaoFen2 dataset. The caption of each subimage displays the corresponding PSNR value. Table 4: The results of ablation experiments on the Landsat8 dataset. Configurations | Proximal | Sharing | Block for | Transpose- | PSNR$\uparrow$ | SSIM$\uparrow$ | SAM$\downarrow$ |  ERGAS$\downarrow$ [t]
---|---|---|---|---|---|---|---|---
Module | Weights |  Eq. (2) | ment |  [b]
I | × | × | × | × | 37.0404 | 0.9498 | 0.0180 |  1.4246 [t]
II | $\checkmark$ | $\checkmark$ | × | × | 38.1650 | 0.9669 | 0.0164 | 1.2943
III | $\checkmark$ | × | $\checkmark$ | × | 38.3213 | 0.9682 | 0.0155 | 1.3215
IV | $\checkmark$ | × | × | $\checkmark$ | 38.5487 | 0.9700 | 0.0150 | 1.2746
GPPNN | $\checkmark$ | × | × | × | 38.9939 | 0.9727 | 0.0138 |  1.2483  [t]

###  4.4 Ablation experiments

To further investigate the role of some modules in the proposed GPPNN, a
series of ablation experiments are carried out. There are 5 different
configurations and the results of ablation experiments are shown in Table 4.

(I) The proximal operators make the current HRMS image restricted to deep
priors. In the first experiment, we delete proximal modules (namely, the
convolutional units in Eqs. (9)&(12d)) to verify the necessity of deep priors.
Table 4 shows that deleting proximal modules make all metrics dramatically get
worse. Therefore, the deep prior plays a significant role in our network.

(II) In the second experiment, we share the weights of all layers. In other
words, the network contains only an MS Block and a PAN block, and the network
is repeatedly fed with the current HRMS image $K$ times. The results in Table
4 demonstrate that sharing the weights will weaken our network’s performance.

(III) As illustrated in Section 3.1, the original problem Eq. (2) is split
into an LRMS-aware subproblem and a PAN-aware subproblem. Now, to verify the
rationality, we generalize Eq. (2) as a neural network with the same
techniques for GPPNN. We exploit this block to build a neural network
corresponding to Eq. (2). From Table 4, we learn that the network for Eq. (2)
is worse than GPPNN. It is necessary to consider two deep priors to separately
account for the generative models of LRMS and PAN images.

(IV) In the last experiment, the convolutional kernel in Eq. (8)/(12c) is
replaced by the kernel in Eq. (6)/(12a) with the rotation of 180° to force
them to satisfy the transposing requirement. It is found that, if the two
kernels transpose to each other, the metrics will slightly become worse. The
reason may be that the model with transposed kernels has fewer degrees of
freedom weakening network’s performance.

##  5 Conclusion and Future Work

This paper provides a new paradigm combining deep unrolling and observation
models of pan-sharpening. We develop a model-driven pan-sharpening network,
GPPNN, by alternatively stacking MS and PAN blocks whose designs are inspired
by two optimization problems. Experiments on three satellites show that our
network outperforms SOTA methods. And a series of ablation experiments verify
the rationality of our network structure.

Remark that each satellite has its unique imaging parameters, including
$\bm{D},\bm{K}$ and $\bm{S}$. GPPNN trained on a satellite cannot be
generalized to another satellite. Hence, the future work is how to improve the
generalization of GPPNN.

## Acknowledgement

This work was supported in part by the National Key Research and Development
Program of China under Grant 2018AAA0102201, and in part by the National
Natural Science Foundation of China under Grants 61976174, 61877049.

## References

  * [1] L. Alparone, B. Aiazzi, S. Baronti, and A. Garzelli.  Sharpening of very high resolution images with spectral distortion minimization.  In IEEE International Geoscience and Remote Sensing Symposium., volume 1, pages 458–460 vol.1, 2003.
  * [2] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang.  Image super-resolution using deep convolutional networks.  IEEE Trans. Pattern Anal. Mach. Intell., 38(2):295–307, 2016\.
  * [3] A. Garzelli, F. Nencini, and L. Capobianco.  Optimal mmse pan sharpening of very high resolution multispectral images.  IEEE Transactions on Geoscience and Remote Sensing, 46(1):228–236, 2008.
  * [4] P. Ghamisi, B. Rasti, N. Yokoya, Q. Wang, B. Hofle, L. Bruzzone, F. Bovolo, M. Chi, K. Anders, R. Gloaguen, P. M. Atkinson, and J. A. Benediktsson.  Multisource and multitemporal data fusion in remote sensing: A comprehensive review of the state of the art.  IEEE Geoscience and Remote Sensing Magazine, 7(1):6–39, 2019.
  * [5] Alan R Gillespie, Anne B Kahle, and Richard E Walker.  Color enhancement of highly correlated images. ii. channel ratio and “chromaticity” transformation techniques.  Remote Sensing of Environment, 22(3):343 – 365, 1987.
  * [6] Karol Gregor and Yann LeCun.  Learning fast approximations of sparse coding.  In ICML, June 21-24, Haifa, Israel, pages 399–406, 2010.
  * [7] R. Haydn, G. W. Dalke, J. Henkel, and J. E. Bare.  Application of the ihs color transform to the processing of multisensor data and image enhancement.  In Proc. Int. Symp. Remote Sensing Arid and Semi-Arid Lands, page 599–616, 1982.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, Las Vegas, NV, USA, June 27-30, pages 770–778, 2016\.
  * [9] L. He, Y. Rao, J. Li, J. Chanussot, A. Plaza, J. Zhu, and B. Li.  Pansharpening via detail injection based convolutional neural networks.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(4):1188–1204, 2019.
  * [10] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In CVPR, Honolulu, HI, USA, July 21-26, pages 2261–2269, 2017\.
  * [11] M. M. Khan, J. Chanussot, L. Condat, and A. Montanvert.  Indusion: Fusion of multispectral and panchromatic images using the induction scaling technique.  IEEE Geoscience and Remote Sensing Letters, 5(1):98–102, 2008.
  * [12] C. A. Laben and B. V. Brower.  Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening.  (U.S. Patent 6011875A), January 2000.
  * [13] Yuelong Li, Mohammad Tofighi, Junyi Geng, Vishal Monga, and Yonina C. Eldar.  Efficient and interpretable deep blind image deblurring via algorithm unrolling.  IEEE Trans. Computational Imaging, 6:666–681, 2020.
  * [14] Junmin Liu, Yunqiao Feng, Changsheng Zhou, and Chunxia Zhang.  Pwnet: An adaptive weight network for the fusion of panchromatic and multispectral images.  Remote. Sens., 12(17):2804, 2020.
  * [15] J. G. Liu.  Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details.  International Journal of Remote Sensing, 21(18):3461–3472, 2000\.
  * [16] Lu Liu, Jun Wang, Erlei Zhang, Bin Li, Xuan Zhu, Yongqin Zhang, and Jinye Peng.  Shallow-deep convolutional network and spectral-discrimination-based detail injection for multispectral imagery pan-sharpening.  IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 13:1772–1783, 2020.
  * [17] P. Liu, L. Xiao, and T. Li.  A variational pan-sharpening method based on spatial fractional-order geometry and spectral–spatial low-rank priors.  IEEE Transactions on Geoscience and Remote Sensing, 56(3):1788–1802, 2018.
  * [18] Xiangyu Liu, Qingjie Liu, and Yunhong Wang.  Remote sensing image fusion based on two-stream fusion network.  Information Fusion, 55:1 – 15, 2020.
  * [19] Lei Ma, Yu Liu, Xueliang Zhang, Yuanxin Ye, Gaofei Yin, and Brian Alan Johnson.  Deep learning in remote sensing applications: A meta-analysis and review.  ISPRS Journal of Photogrammetry and Remote Sensing, 152:166 – 177, 2019.
  * [20] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa.  Pansharpening by convolutional neural networks.  Remote. Sens., 8(7):594, 2016.
  * [21] X. Meng, Y. Xiong, F. Shao, H. Shen, W. Sun, G. Yang, Q. Yuan, r. Fu, and H. Zhang.  A large-scale benchmark data set for evaluating pansharpening performance: Overview and implementation.  IEEE Geoscience and Remote Sensing Magazine, pages 0–0, 2020.
  * [22] Vishal Monga, Yuelong Li, and Yonina C. Eldar.  Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing.  CoRR, abs/1912.10557, 2019.
  * [23] Xavier Otazu, María González-Audícana, Octavi Fors, and Jorge Núñez.  Introduction of sensor spectral response into image fusion methods. application to wavelet-based methods.  IEEE Trans. Geosci. Remote. Sens., 43(10):2376–2385, 2005.
  * [24] F. Ozcelik, U. Alganci, E. Sertel, and G. Unal.  Rethinking cnn-based pansharpening: Guided colorization of panchromatic images via gans.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–16, 2020.
  * [25] R. Tyrrell Rockafellar.  Monotone operators and the proximal point algorithm.  SIAM Journal on Control and Optimization, 14(5):877–898, 1976.
  * [26] R. A. Schowengerdt.  Reconstruction of multispatial multispectral image data using spatial frequency content.  Photogramm Eng Remote Sens, 46(10):1325–1334, 1980.
  * [27] Z. Shao and J. Cai.  Remote sensing image fusion with deep convolutional neural network.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(5):1656–1669, 2018.
  * [28] Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky.  Deep image prior.  In CVPR, Salt Lake City, UT, USA, June 18-22, pages 9446–9454, 2018.
  * [29] G. Vivone, L. Alparone, J. Chanussot, M. Dalla Mura, A. Garzelli, G. A. Licciardi, R. Restaino, and L. Wald.  A critical comparison among pansharpening algorithms.  IEEE Transactions on Geoscience and Remote Sensing, 53(5):2565–2586, 2015.
  * [30] Lucien Wald, Thierry Ranchin, and Marc Mangolini.  Fusion of satellite images of different spatial resolution: Assessing the quality of resulting images.  Photogramm. Eng. Remote Sensing, pages 691–699, 1997.
  * [31] Dong Wang, Ying Li, Li Ma, Zongwen Bai, and Jonathan Cheung-Wai Chan.  Going deeper with densely connected convolutional neural networks for multispectral pansharpening.  Remote. Sens., 11(22):2608, 2019.
  * [32] Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng.  A model-driven deep neural network for single image rain removal.  In CVPR, Seattle, WA, USA, June 13-19, pages 3100–3109, 2020.
  * [33] Na Wang and Jian Sun.  Model meets deep learning in image inverse problems.  CSIAM Transactions on Applied Mathematics, 1(3):365–386, 2020.
  * [34] Yancong Wei, Qiangqiang Yuan, Huanfeng Shen, and Liangpei Zhang.  Boosting the accuracy of multispectral image pansharpening by learning a deep residual network.  IEEE Geosci. Remote. Sens. Lett., 14(10):1795–1799, 2017.
  * [35] Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, and Zongben Xu.  Multispectral and hyperspectral image fusion by MS/HS fusion net.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1585–1594, 2019.
  * [36] Shuang Xu, Ouafa Amira, Junmin Liu, Chun-Xia Zhang, Jiangshe Zhang, and Guanghai Li.  HAM-MFN: hyperspectral and multispectral image multiscale fusion network with RAP loss.  IEEE Trans. Geosci. Remote. Sens., 58(7):4618–4628, 2020.
  * [37] Dong Yang and Jian Sun.  Proximal dehaze-net: A prior learning-based deep network for single image dehazing.  In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII, pages 729–746, 2018.
  * [38] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John W. Paisley.  Pannet: A deep network architecture for pan-sharpening.  In ICCV, Venice, Italy, October 22-29, pages 1753–1761, 2017\.
  * [39] Yan Yang, Jian Sun, Huibin Li, and Zongben Xu.  Admm-csnet: A deep learning approach for image compressive sensing.  IEEE Trans. Pattern Anal. Mach. Intell., 42(3):521–538, 2020\.
  * [40] Q. Yuan, Y. Wei, X. Meng, H. Shen, and L. Zhang.  A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3):978–989, 2018.
  * [41] Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang.  Learning deep CNN denoiser prior for image restoration.  In CVPR, Honolulu, HI, USA, July 21-26, pages 2808–2817, 2017\.