CTP-SCU/2022001, APCTP Pre2022 - 001

# A radiative seesaw model in a supersymmetric modular $A_{4}$ group

Takaaki Nomura<EMAIL_ADDRESS>College of Physics, Sichuan University,
Chengdu 610065, China  Hiroshi Okada<EMAIL_ADDRESS>Asia Pacific
Center for Theoretical Physics (APCTP) - Headquarters San 31, Hyoja-dong, Nam-
gu, Pohang 790-784, Korea

###### Abstract

We propose a supersymmetric radiative seesaw model with modular $A_{4}$
symmetry. Thanks to contributions of supersymmetric partners to one-loop
diagrams generating neutrino masses, we successfully fit neutrino data and
obtain predictions in case of normal hierarchy in a minimal framework that
would not be realized in a non-supersymmetric model. We show a several
predictive figures and demonstrate a best fit benchmark point through
$\chi^{2}$ analysis.

##  I Introduction

The neutrino sector is described by several observables such as neutrino
masses containing two or three non-zero mass eigenvalues; three mixing angles
inducing neutrino oscillations; CP phases including Dirac CP and Majorana
phases that are not yet perfectly confirmed by experiments. We can come up
with various ideas in order not only to reproduce experimental data but also
to predict unknown observables such as phases. The neutrino mass model is
sometimes constructed under a radiatively induced mass scenario Ma:2006km ,
called radiative seesaw, in which we can induce the neutrino mass at loop
levels while tree level mass is forbidden. Such a model would be natural in
order to explain the tiny neutrino masses, retaining not-so-small Yukawa
couplings. It also sometimes accommodates a dark matter (DM) candidate and
potentially causes lepton flavor violation that could leads us to more
intriguing phenomenology. This kind of model usually requires additional
symmetry to guarantee the radiative seesaw mechanism and stability of DM. In
order to get predictions in the lepton sector, non-Abelian discrete flavor
symmetries frequently plays an outstanding role. The symmetries (that has
currently been developed with the help of modular symmetry) would perfectly
match with the radiative seesaw model, because a remnant or part of symmetry
plays a role in replacing the additional symmetry to guarantee the radiative
seesaw mechanism and stability of DM.

The modular flavor symmetry has been proposed by 2017 in ref. Feruglio:2017spp
; deAdelhartToorop:2011re . After that, a lot of ideas have been come up with
in order to realize predictive models. For example, the modular $A_{4}$ flavor
symmetry has been discussed in refs. Feruglio:2017spp ; Criado:2018thu ;
Kobayashi:2018scp ; Okada:2018yrn ; Nomura:2019jxj ; Okada:2019uoy ;
deAnda:2018ecu ; Novichkov:2018yse ; Nomura:2019yft ; Okada:2019mjf ;
Ding:2019zxk ; Nomura:2019lnr ; Kobayashi:2019xvz ; Asaka:2019vev ;
Zhang:2019ngf ; Gui-JunDing:2019wap ; Kobayashi:2019gtp ; Nomura:2019xsb ;
Wang:2019xbo ; Okada:2020dmb ; Okada:2020rjb ; Behera:2020lpd ; Behera:2020sfe
; Nomura:2020opk ; Nomura:2020cog ; Asaka:2020tmo ; Okada:2020ukr ;
Nagao:2020snm ; Okada:2020brs ; Yao:2020qyy ; Chen:2021zty ; Kashav:2021zir ;
Okada:2021qdf ; deMedeirosVarzielas:2021pug ; Nomura:2021yjb ;
Hutauruk:2020xtk ; Ding:2021eva ; Nagao:2021rio ; king ; Okada:2021aoi ;
Nomura:2021pld ; Kobayashi:2021pav ; Dasgupta:2021ggp ; Liu:2021gwa , $S_{3}$
in refs. Kobayashi:2018vbk ; Kobayashi:2018wkl ; Kobayashi:2019rzp ;
Okada:2019xqk ; Mishra:2020gxg ; Du:2020ylx , $S_{4}$ in refs. Penedo:2018nmg
; Novichkov:2018ovf ; Kobayashi:2019mna ; King:2019vhv ; Okada:2019lzv ;
Criado:2019tzk ; Wang:2019ovr ; Zhao:2021jxg ; King:2021fhl ; Ding:2021zbg ;
Zhang:2021olk ; gui-jun ; Nomura:2021ewm , $A_{5}$ in refs. Novichkov:2018nkm
; Ding:2019xna ; Criado:2019tzk , double covering of $A_{4}$ in refs.
Liu:2019khw ; Chen:2020udk ; Li:2021buv , double covering of $S_{4}$ in refs.
Novichkov:2020eep ; Liu:2020akv , and double covering of $A_{5}$ in refs.
Wang:2020lxk ; Yao:2020zml ; Wang:2021mkw ; Behera:2021eut . Other types of
modular symmetries have also been proposed to understand masses, mixings, and
phases of the standard model (SM) in refs. deMedeirosVarzielas:2019cyj ;
Kobayashi:2018bff ; Kikuchi:2020nxn ; Almumin:2021fbk ; Ding:2021iqp ;
Feruglio:2021dte ; Kikuchi:2021ogn ; Novichkov:2021evw ; Kikuchi:2021yog ;
Novichkov:2022wvg . 111Here, we provide useful review references for beginners
Altarelli:2010gt ; Ishimori:2010au ; Ishimori:2012zz ; Hernandez:2012ra ;
King:2013eh ; King:2014nza ; King:2017guk ; Petcov:2017ggy . Different
applications to physics such as dark matter and origin of CP violation are
found in refs. Kobayashi:2021ajl ; Nomura:2019jxj ; Nomura:2019yft ;
Nomura:2019lnr ; Okada:2019lzv ; Baur:2019iai ; Kobayashi:2019uyt ;
Novichkov:2019sqv ; Baur:2019kwi ; Kobayashi:2020hoc ; Tanimoto:2021ehw .
Mathematical study such as possible correction from Kähler potential,
systematic analysis of the fixed points, moduli stabilization are discussed in
refs. Chen:2019ewa ; deMedeirosVarzielas:2020kji ; Ishiguro:2020tmo ;
Abe:2020vmv . It is recently studied that a scenario to derive four-
dimensional modular flavor symmetric models from higher dimensional theory by
assuming the compactification consistent with the modular symmetry
Kikuchi:2022txy .

In this letter, we extend our original radiative seesaw model analyzed under
the non-supersymmetric (non-SUSY) framework Nomura:2019jxj  to the one under
the SUSY version considering contributions from superpartner particles. Thanks
to contributions from supersymmetric partners to diagrams generating neutrino
masses, we successfully fit neutrino data and obtain predictions in case of
normal hierarchy in a minimal framework that would not be realized in a non-
supersymmetric model. We show several predictive figures and demonstrate a
best fit benchmark point through $\chi^{2}$ analysis.

This letter is organized as follows. In Sec. II, we explain our model setup
under the modular $A_{4}$ symmetry, formulating valid mass matrix and their
mixings and the neutrino sector. In Sec III, we carry out our $\chi^{2}$
numerical analysis scanning free parameters. Finally we conclude and discuss
in Sec. IV.

|  $L_{e}$ |  $L_{\mu}$ |  $L_{\tau}$ |  $e^{c}$ |  $\mu^{c}$ |  $\tau^{c}$ |  $N^{c}$ |  $H_{1}$ |  $H_{2}$ |  $\eta_{1}$ |  $\eta_{2}$ |  $\chi$
---|---|---|---|---|---|---|---|---|---|---|---|---
$SU(2)_{L}$ | $\bm{2}$ | $\bm{2}$ | $\bm{2}$ | $\bm{1}$ | $\bm{1}$ | $\bm{1}$ | $\bm{1}$ | $\bm{2}$ | $\bm{2}$ | $\bm{2}$ | $\bm{2}$ | $\bm{1}$
$U(1)_{Y}$ | $-\frac{1}{2}$ | $-\frac{1}{2}$ | $-\frac{1}{2}$ | $1$ | $1$ | $1$ | $0$ | $\frac{1}{2}$ | $-\frac{1}{2}$ | $\frac{1}{2}$ | $-\frac{1}{2}$ | $0$
$A_{4}$ | $1$ | $1^{\prime}$ | $1^{\prime\prime}$ | $1$ | $1^{\prime}$ | $1^{\prime\prime}$ | $3$ | $1$ | $1$ | $1$ | $1$ | $1$
$k$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $-1$ | $0$ | $0$ | $-1$ | $-3$ | $-3$
Table 1: Superfield contents and their charge assignments under
$SU(2)_{L}\times U(1)_{Y}\times A_{4}$, where $-k$ is the number of modular
weight.

##  II Model

In this section we review our model with modular $A_{4}$ symmetry in a
framework of SUSY. We construct the model as minimal assignment as possible.
Thus we give zero modular weight to leptons $L_{(e,\mu,\tau)},\
(e^{c},\mu^{c},\tau^{c})$ and two Higgs doublets $H_{1}$ and $H_{2}$. When we
respectively assign $A_{4}$ singlet representations
$(1,1^{\prime},1^{\prime\prime})$ and $(1,1^{\prime\prime},1^{\prime})$ to
$L_{(e,\mu,\tau)}$ and $(e^{c},\mu^{c},\tau^{c})$ , the charged-lepton mass
matrix is diagonal. Therefore, the observed lepton mixing arises from the
neutrino sector only. To induce neutrino masses at one-loop level minimally,
we introduce the SM singlet superfields $N^{c}$ and $\chi$, and inert doublet
superfields $\eta_{1,2}$. Here, $N^{c}$ is assigned to triplet under $A_{4}$
with $-1$ modular weight and corresponds to heavy Majorana neutrino. The
second inert doublet $\eta_{2}$ plays the same role in generating the mass
terms for $\eta_{1,2}$ as the MSSM Higgs sector $H_{1,2}$. The singlet $\chi$
is required in order to connect the boson loop in neutrino mass generating
diagram. 222SUSY does not allow to induce the interaction
$(H_{1}^{\dagger}\eta_{1})^{2}$ in a renormalizable theory. Superfields
$H_{1,2}$, $\eta_{1,2}$ and $\chi$ are chosen to be trivial singlet under
$A_{4}$, while zero modular weight for $H_{1,2}$ and $(-1,-3,-3)$ weights for
$(\eta_{1},\eta_{2},\chi)$ are assigned; we choose the assignment to make the
model minimal. All the charge assignment and field contents are summarized in
Table 1. Under these symmetries, our valid superpotential is given as follows
(formulas in modular $A_{4}$ framework are summarized in the Appendix):

$\displaystyle{\cal W}$
$\displaystyle=\sum_{\ell=e,\mu,\tau}y_{\ell}\ell^{c}L_{\ell}H_{1}$
$\displaystyle+a_{\eta}(y_{1}N^{c}_{1}+y_{2}N^{c}_{3}+y_{3}N^{c}_{2})L_{e}\eta_{1}+b_{\eta}(y_{2}N^{c}_{2}+y_{1}N^{c}_{3}+y_{3}N^{c}_{1})L_{\mu}\eta_{1}+c_{\eta}(y_{3}N^{c}_{3}+y_{1}N^{c}_{2}+y_{2}N^{c}_{1})L_{\tau}\eta_{1}$
$\displaystyle+m_{N}\left[y_{1}(2N^{c}_{1}N^{c}_{1}-N^{c}_{2}N^{c}_{3}-N^{c}_{3}N^{c}_{2})+y_{2}(2N^{c}_{2}N^{c}_{2}-N^{c}_{1}N^{c}_{3}-N^{c}_{3}N^{c}_{1})+y_{3}(2N^{c}_{3}N^{c}_{3}-N^{c}_{1}N^{c}_{2}-N^{c}_{2}N^{c}_{1})\right]$
$\displaystyle+\mu_{1}Y^{(6)}_{1}H_{1}\eta_{2}\chi+\mu_{2}Y^{(4)}_{1}H_{2}\eta_{1}\chi+\mu_{\chi}Y^{(6)}_{1}\chi\chi+\mu_{H}H_{1}H_{2}+\mu_{\eta}Y^{(4)}_{1}\eta_{1}\eta_{2},$
(II.1)

where $Y^{(2)}_{3}\equiv(y_{1},y_{2},y_{3})^{T}$, and we imposed R-parity
forbidding the R-parity violating terms such as $L_{e}H_{1},\
L_{e}L_{\mu}\tau^{c}$. Note here that oddness of modular weight provides
oddness under accidental $Z_{2}$ symmetry since all modular forms have even
modular weight in modular $A_{4}$ framework and sum of modular weight of
superfields should be even to make modular weight of a term zero.

Then, the valid renormalizable soft SUSY breaking Lagrangian is found as
follows:

$\displaystyle-{\cal L}_{\rm soft}$
$\displaystyle=A_{1}Y^{(6)}_{1}H_{1}\eta_{2}\chi+A_{2}Y^{(4)}_{1}H_{2}\eta_{1}\chi+m^{2}_{H_{1}}|H_{1}|^{2}+m^{2}_{H_{2}}|H_{2}|^{2}$
$\displaystyle+B_{\tilde{N}}^{2}\Bigl{[}y_{1}(2\tilde{N}^{c}_{1}\tilde{N}^{c}_{1}-\tilde{N}^{c}_{2}\tilde{N}^{c}_{3}-\tilde{N}^{c}_{3}\tilde{N}^{c}_{2})+y_{2}(2\tilde{N}^{c}_{2}\tilde{N}^{c}_{2}-\tilde{N}^{c}_{1}\tilde{N}^{c}_{3}-\tilde{N}^{c}_{3}\tilde{N}^{c}_{1})$
$\displaystyle\qquad\quad+y_{3}(2\tilde{N}^{c}_{3}\tilde{N}^{c}_{3}-\tilde{N}^{c}_{1}\tilde{N}^{c}_{2}-\tilde{N}^{c}_{2}\tilde{N}^{c}_{1})\Bigr{]}$
$\displaystyle+B_{H}^{2}H_{1}H_{2}+B_{\eta}^{2}Y^{(4)}_{1}\eta_{1}\eta_{2}+B_{2}^{2}Y^{(6)}_{1}\chi\chi,$
(II.2)

where all fields are supposed to be scalars; $\tilde{N}^{c}$’s are gauge
singlet sneutrinos. Note that we also have SUSY breaking terms associated with
sleptons, squarks and gauginos in the MSSM. In our analysis, however, we do
not discuss super partner of the SM fermions and gauge bosons assuming they
are heavy enough to avoid experimental constraints, and focus on neutrino mass
generating mechanism.

###  II.1 $N^{c}$ mass matrix

The mass matrix of right-handed neutral fermions is straightforwardly found
from the suerpotential as follows:

$\displaystyle{\cal M_{N}}$
$\displaystyle=m_{N}\left[\begin{array}[]{ccc}2y_{1}&-y_{3}&-y_{2}\\\
-y_{3}&2y_{2}&-y_{1}\\\ -y_{2}&-y_{1}&2y_{3}\\\ \end{array}\right]\equiv
m_{N}{\cal Y},$ (II.6)

where ${\cal M_{N}}$ is diagonalized by $D_{N}\equiv U_{N}{\cal
M_{N}}U_{N}^{T}\equiv(M_{1},M_{2},M_{3})$, where $U_{N}$ is a unitary matrix,
and we define the mass eigen-field $\psi^{c}$ as $N^{c}\equiv
U^{T}_{N}\psi^{c}_{N}$.

###  II.2 $\tilde{N}^{c}$ mass matrix

The mass matrix of $\tilde{N}^{c}$ arises from the $F$ and soft SUSY breaking
terms;

$\displaystyle-{\cal L}_{\tilde{N}}=|m_{N}|^{2}(\tilde{N}^{c})^{*}{\cal
Y}^{*}{\cal Y}\tilde{N}^{c}+B_{\tilde{N}}^{2}\tilde{N}^{c}{\cal
Y}\tilde{N}^{c}+{\rm h.c.}.$ (II.7)

Here, we write $\tilde{N}^{c}_{i}\equiv(\phi_{R}+i\phi_{I})_{i}/\sqrt{2}$.
Then, mass terms are given by

$\displaystyle-{\cal L}_{\tilde{N}}$
$\displaystyle=\phi_{R_{i}}^{T}\left(\frac{|m_{N}|^{2}}{2}{\cal Y}^{*}{\cal
Y}+B_{\tilde{N}}^{2}{\cal Y}+{\rm
h.c.}\right)_{ij}\phi_{R_{j}}+\phi_{I_{i}}^{T}\left(\frac{|m_{N}|^{2}}{2}{\cal
Y}^{*}{\cal Y}-B_{\tilde{N}}^{2}{\cal Y}+{\rm h.c.}\right)_{ij}\phi_{I_{j}}$
$\displaystyle\equiv\phi_{R_{i}}^{T}M_{R_{ij}}^{2}\phi_{R_{j}}+\phi_{I_{i}}^{T}M_{I_{ij}}^{2}\phi_{I_{j}}.$
(II.8)

The mass matrix $M_{R(I)}^{2}$ is diagonalized such that $D_{R(I)}^{2}\equiv
O_{R(I)}M_{R(I)}^{2}O_{R(I)}^{T}\equiv(m^{2}_{R(I)_{1}},m^{2}_{R(I)_{2}},m^{2}_{R(I)_{3}})$,
where $O_{R(I)}$ is an orthogonal matrix. Furthermore, we define the mass
eigen-fields $\tilde{n}_{R,I}$ then we have
$\phi_{R(I)}=O^{T}_{R(I)}\tilde{n}_{R(I)}$.

###  II.3 $\chi-\eta$ mixing

Scalar fields $\chi$ and $\eta_{1,2}$ mix each other due to the soft-breaking
terms associated with $A_{1}$ and $A_{2}$ after the EW symmetry breaking by
non-zero VEV of $H_{1,2}$. Here, we assume the mixing between $\chi$ and
$\eta_{1}$ is only active for simplicity. Note, however, that this assumption
does not affect the mechanism of neutrino mass matrix.
Scalar boson mixing is then defined by

$\displaystyle\chi_{R}=c_{R}H_{1}-s_{R}H_{2},\quad\eta_{R_{1}}=s_{R}H_{1}+c_{R}H_{2},$
$\displaystyle\chi_{I}=c_{I}A_{1}-s_{I}A_{2},\quad\eta_{I_{1}}=s_{I}A_{1}+c_{I}A_{2},$
(II.9)

where $s_{R/I}(c_{R/I})$ is the short-hand symbol of
$\sin\theta_{R/I}(\cos\theta_{R/I})$, and $H_{1,2}$ and $A_{1,2}$ are
respectively CP-even and odd mass eigenstates.

We also consider the mixing between the fermionic superpartners of $\chi$ and
$\eta$ denoted by $\tilde{\chi}$ and $\tilde{\eta}_{1}$. Fermion mixing is
defined by

$\displaystyle\tilde{\chi}=c_{\alpha}\psi_{\chi}-s_{\alpha}\psi_{\eta},\quad\tilde{\eta}_{1}=s_{\alpha}\psi_{\chi}+c_{\alpha}\psi_{\eta},$
(II.10)

where $s_{\alpha}(c_{\alpha})$ is the short-hand symbol of
$\sin\theta_{\alpha}(\cos\theta_{\alpha})$, and $\psi_{\chi}$ and
$\psi_{\eta}$ are mass eigenstates.

Figure 1: One loop diagrams generating neutrino mass where we write particles
inside loops by flavor eigenstates.

###  II.4 Neutrino sector

Now we can discuss the neutrino sector estimating neutrino mass at one-loop
level. Our valid renormalizable Lagrangian is explicitly given by

$\displaystyle-{\cal L}^{\nu}$
$\displaystyle=\frac{1}{\sqrt{2}}(\psi^{c}_{N})^{T}_{i}(U_{N})_{ia}Y_{ab}\nu_{b}(s_{R}H_{1}+c_{R}H_{2})+\frac{i}{\sqrt{2}}(\psi^{c}_{N})^{T}_{i}(U_{N})_{ia}Y_{ab}\nu_{b}(s_{I}A_{1}+c_{I}A_{2})$
$\displaystyle+\frac{1}{\sqrt{2}}(\tilde{n}_{R})^{T}_{i}(O_{R})_{ia}Y_{ab}\nu_{b}(s_{\alpha}\psi_{\chi}+c_{\alpha}\psi_{\eta})+\frac{i}{\sqrt{2}}(\tilde{n}_{I})^{T}_{i}(O_{I})_{ia}Y_{ab}\nu_{b}(s_{\alpha}\psi_{\chi}+c_{\alpha}\psi_{\eta})+{\rm
h.c.}.$ (II.11)

These interactions induce neutrino mass at one-loop level via the diagrams in
Fig. 1. Calculating the diagrams the neutrino mass matrix is given by
$m_{\nu}\equiv m_{\nu}^{(I)}+m_{\nu}^{(II)}$;

$\displaystyle(m_{\nu}^{(I)})_{ab}$
$\displaystyle=-\frac{1}{2(4\pi)^{2}}Y^{T}_{a\beta}(U^{T}_{N})_{\beta
i}D_{N_{i}}(U_{N})_{i\alpha}Y_{\alpha b}$
$\displaystyle\times\left[s^{2}_{R}F(m_{H_{1}},M_{i})-s^{2}_{I}F(m_{A_{1}},M_{i})+c^{2}_{R}F(m_{H_{2}},M_{i})-c^{2}_{I}F(m_{A_{2}},M_{i})\right],$
(II.12) $\displaystyle(m_{\nu}^{(II)})_{ab}$
$\displaystyle=-\frac{1}{2(4\pi)^{2}}Y^{T}_{a\beta}Y_{\alpha b}$
$\displaystyle\times\left[M_{\psi_{\chi}}s^{2}_{\alpha}[(O_{R})_{\beta
i}F(m_{R_{i}},M_{\psi_{\chi}})(O_{R})^{T}_{i\alpha}-(O_{I})_{\beta
i}F(m_{I_{i}},M_{\psi_{\chi}})(O_{I})^{T}_{i\alpha}]\right.$
$\displaystyle\left.+M_{\psi_{\eta}}c^{2}_{\alpha}[(O_{R})_{\beta
i}F(m_{R_{i}},M_{\psi_{\eta}})(O_{R})^{T}_{i\alpha}-(O_{I})_{\beta
i}F(m_{I_{i}},M_{\psi_{\eta}})(O_{I})^{T}_{i\alpha}]\right],$ (II.13)
$\displaystyle F(m_{a},m_{b})$
$\displaystyle=\int_{0}^{x}dx\ln\left[x\left(\frac{m_{a}^{2}}{m_{b}^{2}}-1\right)+1\right],$
(II.14)

where $(m_{\nu}^{(I)})_{ab}$ and $(m_{\nu}^{(II)})_{ab}$ denote contribution
from diagram (I) and (II) respectively. Here, $(m_{\nu}^{(I)})_{ab}$ comes
from the first line of Eq.(II.11), while $(m_{\nu}^{(II)})_{ab}$ comes from
the second line of Eq.(II.11). Then, $m_{\nu}$ is diagonalized by a unitary
matrix $U_{\rm PMNS}$; $U_{\rm PMNS}^{T}m_{\nu}U_{\rm PMNS}\equiv{\rm
diag}(m_{1},m_{2},m_{3})$.

Several experimental data are given as follows. We write mass square
difference

$\displaystyle(\mathrm{NH}):\ \Delta m_{\rm
atm}^{2}=m_{3}^{2}-m_{1}^{2},\quad(\mathrm{IH}):\ \Delta m_{\rm
atm}^{2}=m_{2}^{2}-m_{3}^{2},$ (II.15)

where $\Delta m_{\rm atm}^{2}$ is atmospheric neutrino mass square difference,
and NH and IH represent the normal hierarchy and the inverted hierarchy,
respectively. Solar mass square difference is given as follows:

$\displaystyle\Delta m_{\rm sol}^{2}=m_{2}^{2}-m_{1}^{2},$ (II.16)

which can be compared to the observed value. $U_{\rm PMNS}$ is parametrized by
three mixing angle $\theta_{ij}(i,j=1,2,3;i<j)$, one CP violating Dirac phase
$\delta_{CP}$, and two Majorana phases $\\{\alpha_{21},\alpha_{32}\\}$ as
follows:

$U_{\rm
PMNS}=\begin{pmatrix}c_{12}c_{13}&s_{12}c_{13}&s_{13}e^{-i\delta_{CP}}\\\
-s_{12}c_{23}-c_{12}s_{23}s_{13}e^{i\delta_{CP}}&c_{12}c_{23}-s_{12}s_{23}s_{13}e^{i\delta_{CP}}&s_{23}c_{13}\\\
s_{12}s_{23}-c_{12}c_{23}s_{13}e^{i\delta_{CP}}&-c_{12}s_{23}-s_{12}c_{23}s_{13}e^{i\delta_{CP}}&c_{23}c_{13}\end{pmatrix}\begin{pmatrix}1&0&0\\\
0&e^{i\frac{\alpha_{21}}{2}}&0\\\
0&0&e^{i\frac{\alpha_{31}}{2}}\end{pmatrix},$ (II.17)

where $c_{ij}$ and $s_{ij}$ stands for $\cos\theta_{ij}$ and $\sin\theta_{ij}$
respectively. Then, each of mixing is given in terms of the component of
$U_{\mathrm{PMNS}}$ as follows:

$\displaystyle\sin^{2}\theta_{13}=|(U_{\mathrm{PMNS}})_{13}|^{2},\quad\sin^{2}\theta_{23}=\frac{|(U_{\mathrm{PMNS}})_{23}|^{2}}{1-|(U_{\mathrm{PMNS}})_{13}|^{2}},\quad\sin^{2}\theta_{12}=\frac{|(U_{\mathrm{PMNS}})_{12}|^{2}}{1-|(U_{\mathrm{PMNS}})_{13}|^{2}}.$
(II.18)

Also, we compute the Jarlskog invariant, $\delta_{CP}$ derived from PMNS
matrix elements $U_{\alpha i}$:

$J_{CP}=\text{Im}[U_{e1}U_{\mu 2}U_{e2}^{*}U_{\mu
1}^{*}]=s_{23}c_{23}s_{12}c_{12}s_{13}c^{2}_{13}\sin\delta_{CP},$ (II.19)

and the Majorana phases are also estimated in terms of other invariants
$I_{1}$ and $I_{2}$:

$I_{1}=\text{Im}[U^{*}_{e1}U_{e2}]=c_{12}s_{12}c_{13}^{2}\sin\left(\frac{\alpha_{21}}{2}\right),\
I_{2}=\text{Im}[U^{*}_{e1}U_{e3}]=c_{12}s_{13}c_{13}\sin\left(\frac{\alpha_{31}}{2}-\delta_{CP}\right).$
(II.20)

In addition, the effective mass for the neutrinoless double beta decay is
given by

$\displaystyle\langle
m_{ee}\rangle=|m_{1}\cos^{2}\theta_{12}\cos^{2}\theta_{13}+m_{2}\sin^{2}\theta_{12}\cos^{2}\theta_{13}e^{i\alpha_{21}}+m_{3}\sin^{2}\theta_{13}e^{i(\alpha_{31}-2\delta_{CP})}|,$
(II.21)

where its observed value could be measured by KamLAND-Zen in future KamLAND-
Zen:2016pfg . We will adopt the neutrino experimental data in NuFit5.0
Esteban:2018azc  in oder to perform the numerical $\chi^{2}$ analysis.

##  III Numerical analysis

In this section, we show our numerical $\chi^{2}$ analysis where we set the
following ranges for free parameters,

$\displaystyle\\{a_{\eta},b_{\eta},c_{\eta},s_{R},s_{I},s_{\alpha}\\}\in[10^{-3},1],$
(III.1)
$\displaystyle\\{m_{N},B_{\tilde{N}},m_{H_{1}},m_{H_{2}},m_{A_{1}},m_{A_{2}},M_{\psi_{\chi}},M_{\psi_{\eta}}\\}\in[10^{2},10^{5}]{\rm
GeV},$ (III.2)

where modulus $\tau$ runs over the fundamental region. Notice here that we
focus on the case of NH because we find it difficult to obtain the allowed
region within $\sqrt{\chi^{2}}=6$ in the IH case.

Figure 2: Allowed region of $\tau$ in NH, where the color of points
corresponds to the range of $\sqrt{\chi^{2}}$ value such that blue:
$\sqrt{\chi^{2}}\leq 1$, green: $1<\sqrt{\chi^{2}}\leq 2$, yellow:
$2<\sqrt{\chi^{2}}\leq 3$, and red: $3<\sqrt{\chi^{2}}\leq 5$.

Fig. 2 shows the allowed region of $\tau$. We find that $\tau\simeq
1.49i-1.62i$ within $\sqrt{\chi^{2}}=5$, which would be nearby at a fixed
point of $\tau=i\times\infty$. Here, the color of points corresponds to the
range of $\sqrt{\chi^{2}}$ value such that blue: $\sqrt{\chi^{2}}\leq 1$,
green: $1<\sqrt{\chi^{2}}\leq 2$, yellow: $2<\sqrt{\chi^{2}}\leq 3$, and red:
$3<\sqrt{\chi^{2}}\leq 5$.

Figure 3: Allowed region of $\sum m_{i}$ and Dirac CP phase $\delta_{\rm CP}$.
The color legend is the same as Fig.1.

Fig. 3 shows the allowed region of $\sum m_{i}$ and Dirac CP phase
$\delta_{\rm CP}$. We find that $\delta_{\rm CP}$ is localized at nearby $0$,
then the allowed region of $\sum m_{i}$ is [60-64] meV within
$\sqrt{\chi^{2}}=5$. On the other hand $\delta_{\rm CP}$ is localized at
nearby $\pi$, the allowed region of $\sum m_{i}$ is [65-70] meV within
$\sqrt{\chi^{2}}=5$. Even though we do not show the figure on Majorana phases
$\alpha_{31}$ and $\alpha_{21}$, we obtained localized solutions at nearby $0$
or $\pi$ for $\alpha_{31}$ and $\pi$ for $\alpha_{21}$ only. Note that we have
small CP violation in our allowed parameter region. This is due to small real
part of $\tau$ which is only the source of CP violating phase.

Figure 4: Allowed region of the lightest neutrino mass $m_{1}$ and effective
mass for neutrinoless double beta decay $\langle m_{ee}\rangle$. The color
legend is the same as Fig.1.

Fig. 4 shows the allowed region of the lightest neutrino mass $m_{1}$ and
effective mass for neutrinoless double beta decay $\langle m_{ee}\rangle$.
$m_{1}$ tends to be localized at the range of [2-4] meV within
$\sqrt{\chi^{2}}=5$, while $\langle m_{ee}\rangle$ is localized at nearby
[0.25-0.5] meV within $\sqrt{\chi^{2}}=5$.

| NH
---|---
$\tau$ | $0.000122456+1.58071i$
$[a_{\eta},b_{\eta},c_{\eta}]\times 10^{6}$ | $[95.067,6.17325,9.81507]$
$[m_{N},B_{\tilde{N}}]/{\rm GeV}$ | $[8211.59,302.961]$
$[m_{\psi_{\chi}},m_{\psi_{\eta}}]/{\rm GeV}$ | $[6075.09,348.251]$
$[m_{H_{1}},m_{A_{1}},m_{H_{2}},m_{A_{2}}]/{\rm GeV}$ | $[7639.91,384.573,76.5929,359.136]$
$\Delta m^{2}_{\rm atm}$ | $2.52\times 10^{-3}{\rm eV}^{2}$
$\Delta m^{2}_{\rm sol}$ | $7.48\times 10^{-5}{\rm eV}^{2}$
$\sin\theta_{12}$ | $0.550$
$\sin\theta_{23}$ | $0.759$
$\sin\theta_{13}$ | $0.149$
$[\delta_{CP}^{\ell},\ \alpha_{21},\,\alpha_{31}]$ | $[0.806^{\circ},\,180^{\circ},\,0.633^{\circ}]$
$\sum m_{i}$ |  $62.2$ meV
$\langle m_{ee}\rangle$ |  $0.330$ meV
$\sqrt{\Delta\chi^{2}}$ | $0.536$
Table 2: Numerical benchmark point of our input parameters and observables at
nearby the fixed point $\tau=i\times\infty$ in NH. Here, the NH1 is taken such
that $\sqrt{\Delta\chi^{2}}$ is minimum. On the other hand, NH2 is taken so
that $\delta_{CP}$ is closest to the BF value of $195^{\circ}$ within
$\sqrt{\Delta\chi^{2}}\leq 2$.

Finally, we show a benchmark point for NH in Table 2 that provide minimum
$\sqrt{\chi^{2}}$ in our numerical analysis.

##  IV Conclusion and discussion

We have proposed a supersymmetric radiative seesaw model with modular $A_{4}$
symmetry. Thanks to contributions of the SUSY partners, especially
$\tilde{N}^{c}$, to the diagrams generating neutrino mass matrix, we have
successfully constructed a predictive model in a minimum manner where we would
not have obtained it in a non-SUSY model. Through our $\chi^{2}$ numerical
analysis, we have obtained several allowed regions; $\tau$ is localized at
nearby $\tau\simeq 1.49i-1.62i$ within $\sqrt{\chi^{2}}=5$, which would be
nearby at a fixed point of $\tau=i\times\infty$. We find that $\delta_{\rm
CP}$ is localized at nearby $0$ when the allowed region of $\sum m_{i}$ is
[60-64] meV within $\sqrt{\chi^{2}}=5$ while $\delta_{\rm CP}$ is localized at
nearby $\pi$ when the allowed region of $\sum m_{i}$ is [65-70] meV within
$\sqrt{\chi^{2}}=5$. Even though we do not show the figure on Majorana phases
$\alpha_{31}$ and $\alpha_{21}$, we have obtained localized solutions at
nearby $0$ or $\pi$ for $\alpha_{31}$ and $\pi$ for $\alpha_{21}$ only.
$m_{1}$ tends to be localized at the range of [2-4] meV within
$\sqrt{\chi^{2}}=5$, while $\langle m_{ee}\rangle$ is localized at nearby
[0.25-0.5] meV within $\sqrt{\chi^{2}}=5$.

It would be worthwhile briefly mentioning the other aspects such as lepton
flavor violations (LFVs) and dark matter(DM) candidate. Since input Yukawa
couplings are typically of the order $10^{-6}$ and masses of particle
mediating LFVs are of the order $100$ GeV at the lightest, the typical
branching ratio of the $\mu\to e\gamma$, which gives the most stringent
constraint, is less than $10^{-20}$. Therefore, it is totally safe for these
kinds of constraints, because the upper bounds of $\mu\to e\gamma$ is
$4.2\times 10^{-13}$. Notice here that we do not discuss LFVs induced by soft
SUSY breaking terms in the MSSM assuming these couplings are small enough
since it is beyond our scope.

We have several DM candidates such as the lightest $N^{c}$, $\eta_{1,2},\chi$,
and their super-partners. At first, $N^{c}$ might not be a good DM candidate
since magnitude of associated Yukawa couplings are order of $10^{-6}$ at most,
and the DM annihilation cross section to obtain the relic density would be too
small. Thus, we may need to rely on $\eta_{1,2},\ \chi$ in obtaining observed
relic density. Suppose $\eta_{1,2}$ does not mix with $\chi$, $\eta_{1,2}$
cannot have mass difference between their neutral components in case of
bosons. Thus, $\eta_{1,2}$ would be ruled out as DM candidate from direct
detection experiments since DM-nucleon scattering via running $Z$ boson is
large. Namely, pure $\chi$ would be a good DM candidate. The main annihilation
modes to explain the observed relic density would arise from Higgs potential.
It would be easy to evade from constraints from direct detections via Higgs
portals Kanemura:2010sh . Notice here that we do not discuss a DM analysis
including super-partners that is beyond our scope and will be considered in
future work.

## Acknowledgments

The work of H.O. is supported by the Junior Research Group (JRG) Program at
the Asia-Pacific Center for Theoretical Physics (APCTP) through the Science
and Technology Promotion Fund and Lottery Fund of the Korean Government and
was supported by the Korean Local Governments-Gyeongsangbuk-do Province and
Pohang City. The work is also supported by the Fundamental Research Funds for
the Central Universities (T. N.). H.O. is sincerely grateful for all the KIAS
members.

##  Appendix A Formulas in modular $A_{4}$ framework

Here we summarize some formulas of $A_{4}$ modular symmetry framework. Modular
forms are holomorphic functions of modulus $\tau$, $f(\tau)$, which are
transformed by

$\displaystyle\tau\longrightarrow\gamma\tau=\frac{a\tau+b}{c\tau+d}\
,~{}~{}{\rm where}~{}~{}a,b,c,d\in\mathbb{Z}~{}~{}{\rm and}~{}~{}ad-
bc=1,~{}~{}{\rm Im}[\tau]>0~{},$ (A.1) $\displaystyle
f(\gamma\tau)=(c\tau+d)^{k}f(\tau)~{},~{}~{}\gamma\in\Gamma(N)~{},$ (A.2)

where $k$ is the so-called as the modular weight.

A superfield $\phi^{(I)}$ is transformed under the modular transformation as

$\phi^{(I)}\to(c\tau+d)^{-k_{I}}\rho^{(I)}(\gamma)\phi^{(I)},$ (A.3)

where $-k_{I}$ is the modular weight and $\rho^{(I)}(\gamma)$ represents an
unitary representation matrix corresponding to $A_{4}$ transformation. Thus
superpotential is invariant if sum of modular weight from fields and modular
form in corresponding term is zero (also it should be invariant under $A_{4}$
and gauge symmetry).

The basis of modular forms is weight 2, $Y^{(2)}_{3}=(y_{1},y_{2},y_{3})$,
transforming as a triplet of $A_{4}$ that is written in terms of the Dedekind
eta-function $\eta(\tau)$ and its derivative Feruglio:2017spp :

$\displaystyle y_{1}(\tau)$ $\displaystyle=$
$\displaystyle\frac{i}{2\pi}\left(\frac{\eta^{\prime}(\tau/3)}{\eta(\tau/3)}+\frac{\eta^{\prime}((\tau+1)/3)}{\eta((\tau+1)/3)}+\frac{\eta^{\prime}((\tau+2)/3)}{\eta((\tau+2)/3)}-\frac{27\eta^{\prime}(3\tau)}{\eta(3\tau)}\right),$
$\displaystyle y_{2}(\tau)$ $\displaystyle=$
$\displaystyle\frac{-i}{\pi}\left(\frac{\eta^{\prime}(\tau/3)}{\eta(\tau/3)}+\omega^{2}\frac{\eta^{\prime}((\tau+1)/3)}{\eta((\tau+1)/3)}+\omega\frac{\eta^{\prime}((\tau+2)/3)}{\eta((\tau+2)/3)}\right),$
(A.4) $\displaystyle y_{3}(\tau)$ $\displaystyle=$
$\displaystyle\frac{-i}{\pi}\left(\frac{\eta^{\prime}(\tau/3)}{\eta(\tau/3)}+\omega\frac{\eta^{\prime}((\tau+1)/3)}{\eta((\tau+1)/3)}+\omega^{2}\frac{\eta^{\prime}((\tau+2)/3)}{\eta((\tau+2)/3)}\right)\,,$
$\displaystyle\eta(\tau)$ $\displaystyle=$ $\displaystyle
q^{1/24}\Pi_{n=1}^{\infty}(1-q^{n}),\quad q=e^{2\pi i\tau},\quad\omega=e^{2\pi
i/3}.$

Modular forms with higher weight can be obtained from $y_{1,2,3}(\tau)$. Some
$A_{4}$ singlet modular forms used in our analysis are summarized as

$Y_{1}^{(4)}=y_{1}^{2}+2y_{1}y_{3},\quad Y^{(6)}_{\bf
1}=y_{1}^{2}+y_{2}^{2}+y_{3}^{2}-3y_{1}y_{2}y_{3},\\\ $ (A.5)

where number in superscript indicates modular weight.

## References

  * (1) E. Ma, Phys. Rev. D 73, 077301 (2006) doi:10.1103/PhysRevD.73.077301 [hep-ph/0601225].
  * (2) F. Feruglio, doi:10.1142/9789813238053_0012 [arXiv:1706.08749 [hep-ph]].
  * (3) R. de Adelhart Toorop, F. Feruglio and C. Hagedorn, Nucl. Phys. B 858, 437-467 (2012) doi:10.1016/j.nuclphysb.2012.01.017 [arXiv:1112.1340 [hep-ph]].
  * (4) J. C. Criado and F. Feruglio, SciPost Phys. 5, no.5, 042 (2018) doi:10.21468/SciPostPhys.5.5.042 [arXiv:1807.01125 [hep-ph]].
  * (5) T. Kobayashi, N. Omoto, Y. Shimizu, K. Takagi, M. Tanimoto and T. H. Tatsuishi, JHEP 11, 196 (2018) doi:10.1007/JHEP11(2018)196 [arXiv:1808.03012 [hep-ph]].
  * (6) H. Okada and M. Tanimoto, Phys. Lett. B 791, 54-61 (2019) doi:10.1016/j.physletb.2019.02.028 [arXiv:1812.09677 [hep-ph]].
  * (7) T. Kobayashi, H. Okada and Y. Orikasa, [arXiv:2111.05674 [hep-ph]].
  * (8) T. Nomura and H. Okada, Phys. Lett. B 797, 134799 (2019) doi:10.1016/j.physletb.2019.134799 [arXiv:1904.03937 [hep-ph]].
  * (9) H. Okada and M. Tanimoto, Eur. Phys. J. C 81, no.1, 52 (2021) doi:10.1140/epjc/s10052-021-08845-y [arXiv:1905.13421 [hep-ph]].
  * (10) F. J. de Anda, S. F. King and E. Perdomo, Phys. Rev. D 101, no.1, 015028 (2020) doi:10.1103/PhysRevD.101.015028 [arXiv:1812.05620 [hep-ph]].
  * (11) P. P. Novichkov, S. T. Petcov and M. Tanimoto, Phys. Lett. B 793, 247-258 (2019) doi:10.1016/j.physletb.2019.04.043 [arXiv:1812.11289 [hep-ph]].
  * (12) T. Nomura and H. Okada, Nucl. Phys. B 966, 115372 (2021) doi:10.1016/j.nuclphysb.2021.115372 [arXiv:1906.03927 [hep-ph]].
  * (13) H. Okada and Y. Orikasa, [arXiv:1907.13520 [hep-ph]].
  * (14) G. J. Ding, S. F. King and X. G. Liu, JHEP 09, 074 (2019) doi:10.1007/JHEP09(2019)074 [arXiv:1907.11714 [hep-ph]].
  * (15) T. Nomura, H. Okada and O. Popov, Phys. Lett. B 803, 135294 (2020) doi:10.1016/j.physletb.2020.135294 [arXiv:1908.07457 [hep-ph]].
  * (16) T. Kobayashi, Y. Shimizu, K. Takagi, M. Tanimoto and T. H. Tatsuishi, Phys. Rev. D 100, no.11, 115045 (2019) [erratum: Phys. Rev. D 101, no.3, 039904 (2020)] doi:10.1103/PhysRevD.100.115045 [arXiv:1909.05139 [hep-ph]].
  * (17) T. Asaka, Y. Heo, T. H. Tatsuishi and T. Yoshida, JHEP 01, 144 (2020) doi:10.1007/JHEP01(2020)144 [arXiv:1909.06520 [hep-ph]].
  * (18) D. Zhang, Nucl. Phys. B 952, 114935 (2020) doi:10.1016/j.nuclphysb.2020.114935 [arXiv:1910.07869 [hep-ph]].
  * (19) G. J. Ding, S. F. King, X. G. Liu and J. N. Lu, JHEP 12, 030 (2019) doi:10.1007/JHEP12(2019)030 [arXiv:1910.03460 [hep-ph]].
  * (20) T. Kobayashi, T. Nomura and T. Shimomura, Phys. Rev. D 102, no.3, 035019 (2020) doi:10.1103/PhysRevD.102.035019 [arXiv:1912.00637 [hep-ph]].
  * (21) T. Nomura, H. Okada and S. Patra, Nucl. Phys. B 967, 115395 (2021) doi:10.1016/j.nuclphysb.2021.115395 [arXiv:1912.00379 [hep-ph]].
  * (22) X. Wang, Nucl. Phys. B 957, 115105 (2020) doi:10.1016/j.nuclphysb.2020.115105 [arXiv:1912.13284 [hep-ph]].
  * (23) H. Okada and Y. Shoji, Nucl. Phys. B 961, 115216 (2020) doi:10.1016/j.nuclphysb.2020.115216 [arXiv:2003.13219 [hep-ph]].
  * (24) H. Okada and M. Tanimoto, [arXiv:2005.00775 [hep-ph]].
  * (25) M. K. Behera, S. Singirala, S. Mishra and R. Mohanta, [arXiv:2009.01806 [hep-ph]].
  * (26) M. K. Behera, S. Mishra, S. Singirala and R. Mohanta, [arXiv:2007.00545 [hep-ph]].
  * (27) T. Nomura and H. Okada, [arXiv:2007.04801 [hep-ph]].
  * (28) T. Nomura and H. Okada, [arXiv:2007.15459 [hep-ph]].
  * (29) T. Asaka, Y. Heo and T. Yoshida, Phys. Lett. B 811, 135956 (2020) doi:10.1016/j.physletb.2020.135956 [arXiv:2009.12120 [hep-ph]].
  * (30) H. Okada and M. Tanimoto, Phys. Rev. D 103, no.1, 015005 (2021) doi:10.1103/PhysRevD.103.015005 [arXiv:2009.14242 [hep-ph]].
  * (31) K. I. Nagao and H. Okada, [arXiv:2010.03348 [hep-ph]].
  * (32) H. Okada and M. Tanimoto, JHEP 03, 010 (2021) doi:10.1007/JHEP03(2021)010 [arXiv:2012.01688 [hep-ph]].
  * (33) C. Y. Yao, J. N. Lu and G. J. Ding, JHEP 05 (2021), 102 doi:10.1007/JHEP05(2021)102 [arXiv:2012.13390 [hep-ph]].
  * (34) P. Chen, G. J. Ding and S. F. King, JHEP 04 (2021), 239 doi:10.1007/JHEP04(2021)239 [arXiv:2101.12724 [hep-ph]].
  * (35) M. Kashav and S. Verma, [arXiv:2103.07207 [hep-ph]].
  * (36) H. Okada, Y. Shimizu, M. Tanimoto and T. Yoshida, [arXiv:2105.14292 [hep-ph]].
  * (37) I. de Medeiros Varzielas and J. Lourenço, [arXiv:2107.04042 [hep-ph]].
  * (38) T. Nomura, H. Okada and Y. Orikasa, [arXiv:2106.12375 [hep-ph]].
  * (39) P. T. P. Hutauruk, D. W. Kang, J. Kim and H. Okada, [arXiv:2012.11156 [hep-ph]].
  * (40) G. J. Ding, S. F. King and J. N. Lu, [arXiv:2108.09655 [hep-ph]].
  * (41) K. I. Nagao and H. Okada, [arXiv:2108.09984 [hep-ph]].
  * (42) Georgianna Charalampous, Stephen F. King, George K. Leontaris, Ye-Ling Zhou [arXiv:2109.11379 [hep-ph]].
  * (43) H. Okada and Y. h. Qi, [arXiv:2109.13779 [hep-ph]].
  * (44) T. Nomura, H. Okada and Y. h. Qi, [arXiv:2111.10944 [hep-ph]].
  * (45) T. Kobayashi, H. Otsuka, M. Tanimoto and K. Yamamoto, [arXiv:2112.00493 [hep-ph]].
  * (46) A. Dasgupta, T. Nomura, H. Okada, O. Popov and M. Tanimoto, [arXiv:2111.06898 [hep-ph]].
  * (47) X. G. Liu and G. J. Ding, [arXiv:2112.14761 [hep-ph]].
  * (48) T. Kobayashi, K. Tanaka and T. H. Tatsuishi, Phys. Rev. D 98, no.1, 016004 (2018) doi:10.1103/PhysRevD.98.016004 [arXiv:1803.10391 [hep-ph]].
  * (49) T. Kobayashi, Y. Shimizu, K. Takagi, M. Tanimoto, T. H. Tatsuishi and H. Uchida, Phys. Lett. B 794, 114-121 (2019) doi:10.1016/j.physletb.2019.05.034 [arXiv:1812.11072 [hep-ph]].
  * (50) T. Kobayashi, Y. Shimizu, K. Takagi, M. Tanimoto and T. H. Tatsuishi, PTEP 2020, no.5, 053B05 (2020) doi:10.1093/ptep/ptaa055 [arXiv:1906.10341 [hep-ph]].
  * (51) H. Okada and Y. Orikasa, Phys. Rev. D 100, no.11, 115037 (2019) doi:10.1103/PhysRevD.100.115037 [arXiv:1907.04716 [hep-ph]].
  * (52) S. Mishra, [arXiv:2008.02095 [hep-ph]].
  * (53) X. Du and F. Wang, JHEP 02, 221 (2021) doi:10.1007/JHEP02(2021)221 [arXiv:2012.01397 [hep-ph]].
  * (54) J. T. Penedo and S. T. Petcov, Nucl. Phys. B 939, 292-307 (2019) doi:10.1016/j.nuclphysb.2018.12.016 [arXiv:1806.11040 [hep-ph]].
  * (55) P. P. Novichkov, J. T. Penedo, S. T. Petcov and A. V. Titov, JHEP 04, 005 (2019) doi:10.1007/JHEP04(2019)005 [arXiv:1811.04933 [hep-ph]].
  * (56) T. Kobayashi, Y. Shimizu, K. Takagi, M. Tanimoto and T. H. Tatsuishi, JHEP 02, 097 (2020) doi:10.1007/JHEP02(2020)097 [arXiv:1907.09141 [hep-ph]].
  * (57) S. F. King and Y. L. Zhou, Phys. Rev. D 101, no.1, 015001 (2020) doi:10.1103/PhysRevD.101.015001 [arXiv:1908.02770 [hep-ph]].
  * (58) H. Okada and Y. Orikasa, [arXiv:1908.08409 [hep-ph]].
  * (59) J. C. Criado, F. Feruglio and S. J. D. King, JHEP 02, 001 (2020) doi:10.1007/JHEP02(2020)001 [arXiv:1908.11867 [hep-ph]].
  * (60) X. Wang and S. Zhou, JHEP 05, 017 (2020) doi:10.1007/JHEP05(2020)017 [arXiv:1910.09473 [hep-ph]].
  * (61) Y. Zhao and H. H. Zhang, JHEP 03 (2021), 002 doi:10.1007/JHEP03(2021)002 [arXiv:2101.02266 [hep-ph]].
  * (62) S. F. King and Y. L. Zhou, JHEP 04 (2021), 291 doi:10.1007/JHEP04(2021)291 [arXiv:2103.02633 [hep-ph]].
  * (63) G. J. Ding, S. F. King and C. Y. Yao, [arXiv:2103.16311 [hep-ph]].
  * (64) X. Zhang and S. Zhou, [arXiv:2106.03433 [hep-ph]].
  * (65) Bu-Yao Qu, Xiang-Gan Liu, Ping-Tao Chen, Gui-Jun Ding [arXiv:2106.11659 [hep-ph]].
  * (66) T. Nomura and H. Okada, [arXiv:2109.04157 [hep-ph]].
  * (67) P. P. Novichkov, J. T. Penedo, S. T. Petcov and A. V. Titov, JHEP 04, 174 (2019) doi:10.1007/JHEP04(2019)174 [arXiv:1812.02158 [hep-ph]].
  * (68) G. J. Ding, S. F. King and X. G. Liu, Phys. Rev. D 100, no.11, 115005 (2019) doi:10.1103/PhysRevD.100.115005 [arXiv:1903.12588 [hep-ph]].
  * (69) X. G. Liu and G. J. Ding, JHEP 08, 134 (2019) doi:10.1007/JHEP08(2019)134 [arXiv:1907.01488 [hep-ph]].
  * (70) P. Chen, G. J. Ding, J. N. Lu and J. W. F. Valle, Phys. Rev. D 102, no.9, 095014 (2020) doi:10.1103/PhysRevD.102.095014 [arXiv:2003.02734 [hep-ph]].
  * (71) C. C. Li, X. G. Liu and G. J. Ding, [arXiv:2108.02181 [hep-ph]].
  * (72) P. P. Novichkov, J. T. Penedo and S. T. Petcov, Nucl. Phys. B 963, 115301 (2021) doi:10.1016/j.nuclphysb.2020.115301 [arXiv:2006.03058 [hep-ph]].
  * (73) X. G. Liu, C. Y. Yao and G. J. Ding, Phys. Rev. D 103, no.5, 056013 (2021) doi:10.1103/PhysRevD.103.056013 [arXiv:2006.10722 [hep-ph]].
  * (74) X. Wang, B. Yu and S. Zhou, Phys. Rev. D 103, no.7, 076005 (2021) doi:10.1103/PhysRevD.103.076005 [arXiv:2010.10159 [hep-ph]].
  * (75) C. Y. Yao, X. G. Liu and G. J. Ding, Phys. Rev. D 103, no.9, 095013 (2021) doi:10.1103/PhysRevD.103.095013 [arXiv:2011.03501 [hep-ph]].
  * (76) X. Wang and S. Zhou, [arXiv:2102.04358 [hep-ph]].
  * (77) M. K. Behera and R. Mohanta, [arXiv:2108.01059 [hep-ph]].
  * (78) I. de Medeiros Varzielas, S. F. King and Y. L. Zhou, Phys. Rev. D 101, no.5, 055033 (2020) doi:10.1103/PhysRevD.101.055033 [arXiv:1906.02208 [hep-ph]].
  * (79) T. Kobayashi and S. Tamba, Phys. Rev. D 99, no.4, 046001 (2019) doi:10.1103/PhysRevD.99.046001 [arXiv:1811.11384 [hep-th]].
  * (80) S. Kikuchi, T. Kobayashi, H. Otsuka, S. Takada and H. Uchida, JHEP 11, 101 (2020) doi:10.1007/JHEP11(2020)101 [arXiv:2007.06188 [hep-th]].
  * (81) Y. Almumin, M. C. Chen, V. Knapp-Pérez, S. Ramos-Sánchez, M. Ratz and S. Shukla, JHEP 05 (2021), 078 doi:10.1007/JHEP05(2021)078 [arXiv:2102.11286 [hep-th]].
  * (82) G. J. Ding, F. Feruglio and X. G. Liu, SciPost Phys. 10 (2021), 133 doi:10.21468/SciPostPhys.10.6.133 [arXiv:2102.06716 [hep-ph]].
  * (83) F. Feruglio, V. Gherardi, A. Romanino and A. Titov, JHEP 05 (2021), 242 doi:10.1007/JHEP05(2021)242 [arXiv:2101.08718 [hep-ph]].
  * (84) S. Kikuchi, T. Kobayashi and H. Uchida, [arXiv:2101.00826 [hep-th]].
  * (85) P. P. Novichkov, J. T. Penedo and S. T. Petcov, JHEP 04 (2021), 206 doi:10.1007/JHEP04(2021)206 [arXiv:2102.07488 [hep-ph]].
  * (86) S. Kikuchi, T. Kobayashi, Y. Ogawa and H. Uchida, [arXiv:2112.01680 [hep-ph]].
  * (87) P. P. Novichkov, J. T. Penedo and S. T. Petcov, [arXiv:2201.02020 [hep-ph]].
  * (88) G. Altarelli and F. Feruglio, Rev. Mod. Phys. 82, 2701-2729 (2010) doi:10.1103/RevModPhys.82.2701 [arXiv:1002.0211 [hep-ph]].
  * (89) H. Ishimori, T. Kobayashi, H. Ohki, Y. Shimizu, H. Okada and M. Tanimoto, Prog. Theor. Phys. Suppl. 183, 1-163 (2010) doi:10.1143/PTPS.183.1 [arXiv:1003.3552 [hep-th]].
  * (90) H. Ishimori, T. Kobayashi, H. Ohki, H. Okada, Y. Shimizu and M. Tanimoto, Lect. Notes Phys. 858, 1-227 (2012) doi:10.1007/978-3-642-30805-5
  * (91) D. Hernandez and A. Y. Smirnov, Phys. Rev. D 86, 053014 (2012) doi:10.1103/PhysRevD.86.053014 [arXiv:1204.0445 [hep-ph]].
  * (92) S. F. King and C. Luhn, Rept. Prog. Phys. 76, 056201 (2013) doi:10.1088/0034-4885/76/5/056201 [arXiv:1301.1340 [hep-ph]].
  * (93) S. F. King, A. Merle, S. Morisi, Y. Shimizu and M. Tanimoto, New J. Phys. 16, 045018 (2014) doi:10.1088/1367-2630/16/4/045018 [arXiv:1402.4271 [hep-ph]].
  * (94) S. F. King, Prog. Part. Nucl. Phys. 94, 217-256 (2017) doi:10.1016/j.ppnp.2017.01.003 [arXiv:1701.04413 [hep-ph]].
  * (95) S. T. Petcov, Eur. Phys. J. C 78, no.9, 709 (2018) doi:10.1140/epjc/s10052-018-6158-5 [arXiv:1711.10806 [hep-ph]].
  * (96) A. Baur, H. P. Nilles, A. Trautner and P. K. S. Vaudrevange, Nucl. Phys. B 947, 114737 (2019) doi:10.1016/j.nuclphysb.2019.114737 [arXiv:1908.00805 [hep-th]].
  * (97) T. Kobayashi, Y. Shimizu, K. Takagi, M. Tanimoto, T. H. Tatsuishi and H. Uchida, Phys. Rev. D 101, no.5, 055046 (2020) doi:10.1103/PhysRevD.101.055046 [arXiv:1910.11553 [hep-ph]].
  * (98) P. P. Novichkov, J. T. Penedo, S. T. Petcov and A. V. Titov, JHEP 07, 165 (2019) doi:10.1007/JHEP07(2019)165 [arXiv:1905.11970 [hep-ph]].
  * (99) A. Baur, H. P. Nilles, A. Trautner and P. K. S. Vaudrevange, Phys. Lett. B 795, 7-14 (2019) doi:10.1016/j.physletb.2019.03.066 [arXiv:1901.03251 [hep-th]].
  * (100) T. Kobayashi and H. Otsuka, Phys. Rev. D 101, no.10, 106017 (2020) doi:10.1103/PhysRevD.101.106017 [arXiv:2001.07972 [hep-th]].
  * (101) M. Tanimoto and K. Yamamoto, [arXiv:2106.10919 [hep-ph]].
  * (102) M. C. Chen, S. Ramos-Sánchez and M. Ratz, Phys. Lett. B 801, 135153 (2020) doi:10.1016/j.physletb.2019.135153 [arXiv:1909.06910 [hep-ph]].
  * (103) I. de Medeiros Varzielas, M. Levy and Y. L. Zhou, JHEP 11, 085 (2020) doi:10.1007/JHEP11(2020)085 [arXiv:2008.05329 [hep-ph]].
  * (104) K. Ishiguro, T. Kobayashi and H. Otsuka, JHEP 03, 161 (2021) doi:10.1007/JHEP03(2021)161 [arXiv:2011.09154 [hep-ph]].
  * (105) H. Abe, T. Kobayashi, S. Uemura and J. Yamamoto, Phys. Rev. D 102, no.4, 045005 (2020) doi:10.1103/PhysRevD.102.045005 [arXiv:2003.03512 [hep-th]].
  * (106) S. Kikuchi, T. Kobayashi, H. Otsuka, M. Tanimoto, H. Uchida and K. Yamamoto, [arXiv:2201.04505 [hep-ph]].
  * (107) N. Aghanim et al. [Planck Collaboration], arXiv:1807.06209 [astro-ph.CO].
  * (108) M. Tanabashi et al. (Particle Data Group), Phys. Rev. D 98, 030001 (2018).
  * (109) A. Gando et al. [KamLAND-Zen Collaboration], Phys. Rev. Lett. 117, no. 8, 082503 (2016) Addendum: [Phys. Rev. Lett. 117, no. 10, 109903 (2016)] doi:10.1103/PhysRevLett.117.109903, 10.1103/PhysRevLett.117.082503 [arXiv:1605.02889 [hep-ex]].
  * (110) I. Esteban, M. C. Gonzalez-Garcia, A. Hernandez-Cabezudo, M. Maltoni and T. Schwetz, JHEP 1901, 106 (2019) doi:10.1007/JHEP01(2019)106 [arXiv:1811.05487 [hep-ph]].
  * (111) S. Kanemura, S. Matsumoto, T. Nabeshima and N. Okada, Phys. Rev. D 82, 055026 (2010) doi:10.1103/PhysRevD.82.055026 [arXiv:1005.5651 [hep-ph]].


# Building Segmentation through a Gated Graph Convolutional Neural Network
with Deep Structured Feature Embedding

Yilei Shi<EMAIL_ADDRESS>Qingyu Li<EMAIL_ADDRESS>Xiao Xiang Zhu
<EMAIL_ADDRESS>[ Chair of Remote Sensing Technology, Technical
University of Munich, 80333 Munich, Germany  Signal Processing in Earth
Observation, Technical University of Munich, 80333 Munich, Germany  Remote
Sensing Technology Institute, German Aerospace Center (DLR), Oberpfaffenhofen,
82234 Wessling, Germany

###### Abstract

Automatic building extraction from optical imagery remains a challenge due to,
for example, the complexity of building shapes. Semantic segmentation is an
efficient approach for this task. The latest development in deep convolutional
neural networks (DCNNs) has made accurate pixel-level classification tasks
possible. Yet one central issue remains: the precise delineation of
boundaries. Deep architectures generally fail to produce fine-grained
segmentation with accurate boundaries due to their progressive down-sampling.
Hence, we introduce a generic framework to overcome the issue, integrating the
graph convolutional network (GCN) and deep structured feature embedding (DSFE)
into an end-to-end workflow. Furthermore, instead of using a classic graph
convolutional neural network, we propose a gated graph convolutional network,
which enables the refinement of weak and coarse semantic predictions to
generate sharp borders and fine-grained pixel-level classification. Taking the
semantic segmentation of building footprints as a practical example, we
compared different feature embedding architectures and graph neural networks.
Our proposed framework with the new GCN architecture outperforms state-of-the-
art approaches. Although our main task in this work is building footprint
extraction, the proposed method can be generally applied to other binary or
multi-label segmentation tasks.

###### keywords:

building extraction, semantic segmentation, graph model, gated convoluational
neural networks.

††journal: ISPRS Journal of Photogrammetry and Remote Sensing

url]www.sipeo.bgu.tum.de

##  1 Introduction

Building footprint generation is an active topic in remote sensing field.
Recently, it has received considerable attention due to its huge potential in
autonomous driving, virtual reality, urban planning, environmental, and
demographic applications. Manual extraction of buildings from optical images
is time consuming and difficult in large-scale practice. In contrast, semantic
segmentation is a comparatively inexpensive and time-saving technique for
extracting building footprints. It aims to classify each pixel with a
corresponding class. Various semi-automatic and automatic methods [1] [2] [3]
[4] have been developed to improve segmentation accuracy within this method;
traditionally, feature extraction and classification are its two main steps.
The extraction of such hand-crafted features usually require a strong domain-
specific knowledge.

In recent years, the use of deep learning has garnered great success in
semantic segmentation. In particular, deep convolutional neural networks
(DCNNs) have shown promising results, due to their high capacity for data
learning. DCNNs [5] have instigated compelling advancement over traditional
semantic segmentation methods.

However, exploiting DCNN for semantic segmentation tasks still raises
significant challenges. The convolution layer of a DCNN is a weights sharing
architecture, and it has both shift invariant and spatial invariant
characteristics. While the invariance is clearly desirable for high-level
vision tasks, it may hamper low-level tasks such as pose estimation and
semantic segmentation, where precise localization is required rather than
abstraction of spatial details. For instance, the coarse segmentation output
such as non-sharp boundaries and blob-like shapes is caused by convolution
filters with large receptive fields and pooling layers in DCNN. Moreover, DCNN
fails to fine local details without the consideration of the interactions
between pixels.

To overcome these issues, the probabilistic graph models, such as the
conditional random field (CRF) [7] and Markov random field (MRF) [6], have
been introduced to connect with DCNNs at the final layer. To use CRF for
semantic segmentation, the main concept is to transform the problem of pixel-
wise classification into a problem of probabilistic inference, which assumes
similar pixels should have the same labels. This substantially improves the
predictions of the pixel-wise labels to generate precise borders and
exhaustive segmentation. In [7], instead of using CRF as the post-processing
step, the authors propose an end-to-end architecture that combines the FCN
with a fully connected CRF. However, these frameworks have not sufficiently
extracted the features from the images. Different level features have
different properties for semantic segmentation. Since low-level
characteristics are rich with spatial details but lack semantic information
and high-level characteristics are conversely, they are naturally
complementary. Another issue with CRF is that information propagation is not
sufficient.

In this work, we propose a generic framework for semantic segmentation in this
work, which integrates deep structured feature embedding and the graph
convolutional network. In order to extract more comprehensive and
representative features, we exploit deep structured feature embedding
techniques to enhance the feature fusion by incorporating multi-level
characteristics. Furthermore, we propose a new graph convolutional network,
the gated graph convolutional network (GCN). GCN can aggregate the information
from neighbor nodes (short range), which allows the model to learn about local
structures. A recurrent neural network (RNN) with gated recurrent units (GRUs)
has proven successful to model the long-term dependencies in sequential data.
Hence, we adopt RNN with GRUs for long-range information propagation. The
proposed network integrates the two architectures together, thus taking into
account both local and global contextual dependencies. It is useful for
semantic segmentation tasks. As a consequence, DSFE-GGCN is a trainable end-
to-end framework. We show that joint learning of deep structured feature
embedding and GGCN parameters results in considerable performance gains.

### Contributions

The contributions of this work are summarized as follows:

  * 1.

A generic framework for semantic segmentation is proposed, which integrates
the deep structured feature embedding and a graph convolutional neural network
into an end-to-end workflow.

  * 2.

We propose a novel network architecture, called a “gated graph convolutional
neural network,” which combines the RNN with GRUs for long distance
information propagation and the GCN for short distance information
propagation.

  * 3.

An effective four-step preprocessing approach is proposed for data
augmentation, especially for medium-resolution satellite imagery.

  * 4.

The performance of different DCNNs and the proposed framework is analyzed
through a systematic investigation. Our framework with GGCN surpasses the
state-of-the-art approaches to building footprint extraction.

##  2 Related Work

###  2.1 Semantic segmentation with DCNNs

The fully convolutional network (FCN) was first proposed in [8] for the task
of semantic segmentation, in which convolutional layers take the place of
fully connected layers. FCN makes the training more efficient and the input
size of inference arbitrary. A more memory-efficient approach that used an
alternative decoder variant, SegNet, was proposed in [9]. The stored indices
of the max-pooling step in the downsampling path is used by the decoder for
the operation of upsampling. Another variant of the encoder-decoder
architecture is U-Net [10]. The long skip connections in the network enables
the recovery of the downsample-induced information lost in the encoder.

One key issue for fully convolutional neural networks is that the spatial
resolution is significantly downsampled, which is caused by the operations,
such as strided convolutional layers or pooling layers. In order to overcome
the poor localization property, [11] proposed another approach to improve the
spatial resolution, using a probabilistic graph model CRF to achieve fine-
grained boundaries. Instead of using CRF as a post-processing step, DeepLab-
CRF [7] introduces a fully connected CRF layer, which leads to an end-to-end
trainable network.

###  2.2 Graph model

A graph model is a probabilistic model that encodes a distribution based on a
graph-based representation. The Markov random field (MRF) is one classic graph
model, which uses an undirected graph to describe the joint probability
distribution of random variables. It has been applied to many tasks of image
processing, including image co-registration, image segmentation, and image
super-resolution. MRF takes into account the relationships of the neighbours
to infer the maximal possibility of the pixel’s label. The conditional random
field (CRF) is an extension of MRF, which models the conditional probability
distribution instead of the joint probability distribution. CRF as a
discriminative model shows a better performance when the samples are limited.
The combination of DCNNs and the graph model CRF [11, 7] can produce high-
resolution prediction for better segmentation.

Recent work [23] has extended DCNNs to topologies that differ from the low-
dimensional grid structure. Due to significant computational drawbacks, it is
impractical for real-world use. Henaff et al.[24] and Defferrard et al.[25]
further improve GCN to successfully overcome this issue. The grid-like data
can be interpreted as a special type of graph data, where the node is on the
grid and the number of neighbours is fixed. In this work, we propose a gated
graph convolutional network, which is a trainable inference systems based on
GCN and RNN with GRUs.

###  2.3 Building footprint extraction

Building footprint generation is currently exciting a great deal of interest,
and is an active field of research in the fields of remote sensing,
photogrammetry and computer vision. The established building footprint maps
are used in many important applications to analyze the process of
urbanization, such as urban growth and sustainable urban development.

In [17], the authors propose a multi-stage ConvNet with an upsampling
operation of bilinear interpolation. The trained model achieves a superior
performance on very-high-resolution aerial imagery. Recently, an end-to-end
trainable active contour model (ACM) was developed for building instance
extraction [18], which learns ACM parameterizations using a DCNN. In [19], a
residual refinement network was proposed to extract the building footprint
using aerial images and LiDAR point clouds. In [20], the authors exploit the
improved conditional Wasserstein generative adversarial network to generate
the building footprint automatically. Recent work [21] has shown that most of
the tasks, such as building segmentation, building height estimation, and
building contour extraction, are still difficult for modern convolutional
networks. In this work, we show a significant performance improvement in
building footprint extraction by using our proposed novel framework.

##  3 Methodology

The details of the DSFE-GGCN framework are introduced in this section. The
workflow of the proposed method is shown in Fig. 1. An image can be
generalized as a graph, whose nodes are on the two-dimensional grid. Each
pixel represents a node. The embedding vectors can be computed initially from
node inputs, e.g., node type embeddings, and then propagated on the graph to
aggregate information from the local neighborhood.

Figure 1: An illustration of the proposed DSFE-GGCN framework. The initial
hidden representation of the corresponding node is taken from the feature
vectors in the DSFE step. For a certain time step $t$, the messages from the
neighbourhoods of the node are aggregated by using GCN. After that, the hidden
state of the next time step $t+1$ is updated by gated recurrent units, which
use the hidden state $\mathbf{h}_{i}^{t}$ and the message
$\mathbf{a}_{i}^{t+1}$ at time step $t$ as input. After final timestep $t+n$,
a negative log-likelihood loss function is computed and the whole DSFE-GGCN
model is updated using back-propagation.

###  3.1 Deep structured feature embedding

Deep embedding methods typically map images into an embedding space, where
their distances preserve the relative similarity. In general, the
representations of the data can be learned by graph embedding techniques [22],
which take into account the relationships of the data. In addition, data from
different sources, such as images, point clouds, and social media data, can be
transformed into feature space, which can be further used for segmentation or
other tasks. In this study, the data source is only imagery. Hence, we exploit
a more efficient approach for feature embedding that uses DCNNs as feature
extractor.

However, the resolution of the later layers in the neural network is extremely
downsampled, a phenomenon that is caused by strided convolution, max-polling,
or other operations. Several methods have been introduced to decipher precise
information from the downsampled feature maps. One common approach is to
utilize interpolation techniques [9], which is both computationally cheap and
memory-saving. An alternative is deconvolution, in which recorded indices of
the polling operation are used to retrieve information from the feature maps
[15]. Recently, long skip connections between the contracting and expanding
paths were introduced to retrieve detailed spatial information from the high-
level feature layers [10]. In combination with DenseNet block [13], FC-
DenseNets was proposed in [12], where the upsampling path was composed of
deconvolution, unpooling, and long skip connections. Consequently, all the
feature maps from deconvolution, unpooling, or skip-connections are exploited
for the computation in the upsampling path of the dense blocks. Moreover,
recent work shows that multiple DCNN features extracted from different
networks can be complementary, and could be fused to improve segmentation
accuracy. However, the method for fusing multiple features is still an open
problem that needs systematic investigation.

As mentioned above, low-level features yield better representation of
localization and high-level features can give more comprehensive semantics.
Therefore, in this work, we concatenate different level features progressively
in order to propagate information about localization, semantics, and other
properties through graph convolutional neural networks.

###  3.2 Gated graph convolutional neural network

An undirected and connected graphs $\mathcal{G}=(\mathcal{V},\mathcal{E})$
consists of a set of nodes $\mathcal{V}$ and edges $\mathcal{E}$. The
unnormalized graph Laplacian matrix $\mathbf{L}$ is defined as:

$\mathbf{L}=\mathbf{D}-\mathbf{A},$ (1)

where $\mathbf{A}$ is the adjacency matrix representing the topology of
$\mathcal{G}$, and $\mathbf{D}$ is the degree matrix, which is calculated by
$D_{ii}=\sum_{j}A_{ij}$. The properties of the graph Laplacian $\mathbf{L}$
are symmetric, positive, and semi-defined; therefore the eigenvalue
decomposition can be expressed as:

$\mathbf{L}=\boldsymbol{\Phi}\boldsymbol{\Lambda}\boldsymbol{\Phi}^{T},$ (2)

where $\boldsymbol{\Phi}=\left(\phi_{1},\phi_{2},...,\phi_{n}\right)$ are the
orthonormal eigenvectors, known as the graph Fourier modes, and
$\boldsymbol{\Lambda}=\mathrm{diag}\left(\lambda_{1},\lambda_{2},...,\lambda_{n}\right)$
are the eigenvalues of $\mathbf{L}$, which is a non-negative diagonal matrix.
Assuming a signal $\mathbf{f}$ on the graph nodes $\mathcal{V}$, its graph
Fourier transform can be formulated as
$\hat{\mathbf{f}}=\boldsymbol{\Phi}^{T}\mathbf{f}$. If $\mathbf{g}$ is a
filter, the convolution of $\mathbf{f}$ and $\mathbf{g}$ is written as

$\mathbf{g}*\mathbf{f}=\boldsymbol{\Phi}\left(\left(\boldsymbol{\Phi}^{T}\mathbf{g}\right)\circ\left(\boldsymbol{\Phi}^{T}\mathbf{f}\right)\right)=\boldsymbol{\Phi}\hat{\mathbf{g}}\boldsymbol{\Phi}^{T}\mathbf{f},$
(3)

where $\hat{\mathbf{g}}$ is the spectral representation of the filter. Rather
than computing the Fourier transform $\hat{\mathbf{g}}$, the filter
coefficients can be parameterized as
$\hat{\mathbf{g}}=\sum\limits_{k=0}^{r}\alpha_{k}\beta_{k}$, ask shown in
[24]. With the polynomial parametrization of the filter, the spectral filter
is exactly localized in space. Moreover, the learning complexity is
$\mathcal{O}(r)$, the filter support size, and the same complexity as
classical DCNNs.

In order to avoid explicit multiplication in the spectral domain,
alternatively, the spectral representation $\hat{\mathbf{g}}$ of the filter
$\mathbf{g}$ can be approximated by a Chebyshev polynomial expansion
$g(\boldsymbol{\Lambda})$, which is formulated as:

$g(\boldsymbol{\Lambda})=\sum\limits_{k=0}^{r}\alpha_{k}T_{k}(\tilde{\boldsymbol{\Lambda}}),$
(4)

where $T_{k}(\tilde{\boldsymbol{\Lambda}})$ is the Chebyshev polynomials. The
graph convolution can be defined as:

$\mathbf{g}*\mathbf{f}=\sum\limits_{k=0}^{r}\alpha_{k}T_{k}(\tilde{\mathbf{L}})\mathbf{f},$
(5)

where $\tilde{\mathbf{L}}=2/\lambda_{max}\cdot\mathbf{L}-\mathbf{I}$, and
$\lambda_{max}$ is the maximal eigenvector. In [26], the authors further
simplify the Chebyshev framework, setting $r=1$ and assuming
$\lambda_{max}\approx 2$, allowing them to redefine a single convolutional
layer as simply:

$\mathbf{H}_{i}^{r}=\sigma_{r}\left(\tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}\mathbf{W}\mathbf{H}_{i}^{r-1}\right),$
(6)

where $\mathbf{H}$ is the hidden layer. By taking into account the self-
connections, the original adjacency matrix of the graph $\mathcal{G}$ is
transformed to $\tilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}$, where $\mathbf{I}$
is the identity matrix. $\mathbf{W}$ is the trainable weight matrix and the
new degree matrix $\tilde{\mathbf{D}}$ can be calculated by
$\tilde{D}_{ii}=\sum_{j}\tilde{A}_{ij}$. The function $\sigma_{r}(\cdot)$
denotes a nonlinear activation function. This simplified form improves
computational performance on larger graphs and predictive performance on small
training sets.

#### Propagation model

The propagation process can be formulated as:

$\displaystyle\mathbf{a}_{i}^{t}$
$\displaystyle=\mathcal{M}\left(\mathbf{h}_{j}^{t-1}|j\in\mathcal{V}_{i}\right),$
(7) $\displaystyle\mathbf{h}_{i}^{t}$
$\displaystyle=\mathcal{F}\left(\mathbf{h}_{i}^{t-1},\mathbf{a}_{i}^{t}\right),$
(8)

where $\mathbf{a}_{i}^{t}$ is the message layer at time step $t$, which
represents the messages propagated from its neighbours $\mathcal{V}_{i}$ to
the node $i$. The message layer $\mathbf{a}_{i}^{t}$ at time step $t$ serves
as input to update the hidden layer with function $\mathcal{F}$. Our proposed
method is to use GCN as the message function, which makes it easy for the
propagation model to learn to propagate the node embeddings for node $i$ to
all nodes reachable from $i$. We adopt gating techniques to surpass GCN
performance, because its own memory can be maintained and the valuable
information from neighbours can be gathered with its aid.

The unrolled propagation model at timestep $t$ can be written as:

$\displaystyle\mathbf{a}_{i}^{t}=$
$\displaystyle\sigma_{r}\left(\tilde{\mathbf{D}}^{-1/2}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-1/2}\mathbf{W}\mathbf{h}_{i}^{t-1}\right),$
(9) $\displaystyle\mathbf{r}_{i}^{t}=$
$\displaystyle\sigma_{s}\left(\mathbf{W}_{r}\mathbf{h}_{i}^{t-1}+\mathbf{U}_{r}\mathbf{a}_{i}^{t}\right),$
(10) $\displaystyle\mathbf{z}_{i}^{t}=$
$\displaystyle\sigma_{s}\left(\mathbf{W}_{z}\mathbf{h}_{i}^{t-1}+\mathbf{U}_{z}\mathbf{a}_{i}^{t}\right),$
(11) $\displaystyle\tilde{\mathbf{h}}_{i}^{t}=$
$\displaystyle\tanh\left(\mathbf{W}\left(\mathbf{r}_{i}^{t}\circ\mathbf{h}_{i}^{t-1}\right)+\mathbf{U}\mathbf{a}_{i}^{t}\right),$
(12) $\displaystyle\mathbf{h}_{i}^{t}=$
$\displaystyle\left(1-\mathbf{z}_{i}^{t}\right)\circ\mathbf{h}_{i}^{t-1}+\mathbf{z}_{i}^{t}\circ\tilde{\mathbf{h}}_{i}^{t},$
(13)

where $\mathbf{r}$ and $\mathbf{z}$ are the reset and update gates, and
$\mathbf{W}_{r},\mathbf{W}_{z},\mathbf{U}_{r},\mathbf{U}_{z}$ are learnable
weights for different gates. The function $\sigma_{r}$ is the ReLU function,
$\sigma_{s}$ is the logistic sigmoid function, and $\circ$ is interior
product. The initial hidden representation of the corresponding node is taken
from the feature vectors of the DSFE step. For a certain time step $t$, the
messages from the neighbourhoods of the node are aggregated by using GCN.
After that, the hidden state of next time step $t+1$ is updated by gated
recurrent units, which use the hidden state $\mathbf{h}_{i}^{t}$ and the
message $\mathbf{a}_{i}^{t+1}$ at time step $t$ as input. With the help of the
reset gate and the update gate in GRU [29], the node can maintain its own
memory and extract useful information from incoming messages. Along with the
increase of the time step, it is capable of capturing the long range
dependencies, which has been difficult to model in vanilla GCN.

#### Prediction model

The node classification is defined as:

$p=\mathrm{softmax}\left(\mathbf{h}_{i}^{t+n}\right),$ (14)

Since we have transferred the binary semantic segmentation problem to the
multi-label pixel labeling task, a softmax with negative log-likelihood loss
function is used to predict the probability of each node.

##  4 Experiments

###  4.1 Datasets

In this work, we use Planetscope satellite imagery [30] with three channels
(R, G, B) at a 3 m spatial resolution. The imagery is acquired by Doves, which
can provide complete coverage of Earth once a day. The study sites cover four
cities: (1) Munich, Germany; (2) Rome, Italy; (3) Paris, France; and (4)
Zurich, Switzerland. The corresponding building footprint layer is downloaded
from OpenStreetMap (OSM) [31]. The images are cropped with a patch size of
$\textrm{64}\times\textrm{64}$. The overlap of each patch has 19 pixels in one
direction. At the end, 48,000 sample patches are generated. The training data
has 80% of the patches and the testing data has 20% of the patches. The
training and testing data is spatially separated.

###  4.2 Preprocessing

The datasets utilized in this work consist of Planetscope satellite imagery
and OSM building footprints as ground truth. However, since data sources for
OSM are different from satellite imagery, there are likely inconsistencies
between OSM building footprints and satellite imagery. Therefore, we need to
carry out preprocessing steps to limit the inconsistencies before the
experiments, which include band normalization, coregistration, refinement, and
a truncated signed distance map (TSDM) (see Fig. 2).

Figure 2: Illustration of preprocessing step

In the next section, we will mainly focus on the coregistration and TSDM
steps.

####  4.2.1 Coregistration

One inconsistency is misalignments between OSM building footprints and
satellite imagery, which is caused by different projections and accuracy
levels from data sources. Fig. 3 (a) shows an example of and OSM building
footprint overlaid with the corresponding satellite imagery. There are
noticeable misalignments between the building footprint and the satellite
imagery. These misalignments lead to inaccurate training samples, which need
to be corrected.

(a)

(b)

Figure 3:  (a) Before coregistration; (b) After coregistration.

The coregistration process includes several steps: (1) The satellite imagery
is transformed from RGB to gray scale; (2) The Gaussian gradient of grayscale
imagery is calculated; (3) The cross correlation between the gradient
magnitude of the grayscale image and building footprints is computed; (4) The
pixel with the maximum cross correlation is found and the offset in both row
and column direction can be derived. Fig. 3 (b) shows the result after
coregistration.

####  4.2.2 Truncated signed distance map

In order to incorporate both semantic information about class labels and
geometric properties in the training of the network, the distances of pixels
to boundaries of buildings are extracted as output representations. In our
experiment, the value of the signed-distance function (SDF) is determined by
the distance between the pixel and its nearest point on the boundary. Positive
values imply that the pixels are within the buildings and negative values
indicate the outside of buildings.

Then we truncate the distance at a given threshold to incorporate only the
pixels closest to the border. In this case, the problem in our research is a
multi-label segmentation task, which enhances the result of prediction by the
detailed signed distance map. The truncated signed distance function can be
expressed as:

$D(\mathbf{x})=\delta_{d}\cdot\min\left(\min\limits_{\mathbf{x}\in\mathbf{X}}(d(\mathbf{x})),T_{d}\right),$
(15)

where $\min\limits_{\mathbf{x}\in\mathbf{X}}(d(\mathbf{x}))$ denotes the
euclidean distance $d(\mathbf{x})$ between the pixel and its nearest point on
the boundary of the building. The term $\delta_{d}$ is a sign function with
the implication of inside or outside of objects; $T_{d}$ is the truncated
threshold.

###  4.3 Experimental setup

We use 11 classes for the truncated signed distance map, which is in $[0,10]$
and the truncated threshold is set to 5. For all networks, a stochastic
gradient descent (SGD) is used and the learning rate is set to
$\mathrm{10}^{-4}$. The negative log likelihood loss (NLLLoss) is adopted as
the loss function. The proposed framework is implemented using Pytorch.
Experiments are run on a NVIDIA Tesla P100 16 GB GPU. Several semantic
segmentation methods, which include FCN-32s, SegNet, FCN-16s, U-Net, FCN-8s,
ResNet-DUC, CWGAN-GP, FC-DenseNet, GCN, GraphSAGE, and GGNN, are chosen as the
algorithms of comparison.

###  4.4 Numerical results

The three metrics in the following experiments selected to evaluate the
results are: overall accuracy (OA), F1 scores, and the Intersection over Union
(IoU) scores. The experiments are carried out in following way. First, as a
baseline, we assess the capability of different deep convolutional neural
networks for building footprint extraction. Then, we choose different DCNNs
for deep structured feature embedding and combine it with GCN [26] to decide
which DCNN is the best feature extractor for our proposed framework. At the
end, we use the best feature extractor for DSFE and compare the proposed
framework to different graph models.

####  4.4.1 Baseline with different DCNNs

In this section, the performance of the state-of-the-art DCNNs for building
footprint generation are firstly investigated, which indicates the capability
of each DCNN for feature extraction and precise localization.

|  |  |
---|---|---|---
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

Figure 4: Visualized comparison of the predicted results using different DCNNs. The predicted label is red, which overlays the optical image. (a) FCN-32s; (b) FCN-16s; (c) ResNet-DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-DenseNet; (j) Ground truth.  Methods | OA | F1 | IoU
---|---|---|---
FCN-32s | 0.7318 | 0.2697 | 0.1559
FCN-16s | 0.7698 | 0.3993 | 0.2494
ResNet-DUC | 0.7945 | 0.4542 | 0.2930
E-Net | 0.8243 | 0.5427 | 0.3724
SegNet | 0.8261 | 0.5558 | 0.3848
U-Net | 0.8412 | 0.6043 | 0.4329
FCN-8s | 0.8472 | 0.6222 | 0.4513
CWGAN-GP | 0.8483 | 0.6268 | 0.4562
FC-DenseNet | 0.8551 | 0.6328 | 0.4628
Table 1: Comparison of different deep convolutional neural networks on the
test datasets

FCN-32s and FCN-16s exhibit poor performance, since the feature map of later
layers have only high-level semantics with poor localization. ResNet-DUC can
achieve better result than the previous two because of hybrid dilated
convolution and dense upsampling convolution. However, it is limited due to
the lack of skip connections. Max-pooling indices are reused in SegNet during
the decoding process, which can reduce the parameter number of network leading
to efficient training. However, as it only use indices of max-pooling to
decoder, some local details cannot be recovered, e.g., small buildings will be
neglected. FCN-8s and U-Net outperform previous networks due to the
concatenation of low-level features. Compared to the other CNN models, cwGAN-
gp shows promising results for building footprint generation. The enhancement
of performance is motivated by the min-max competition between the
discriminator and the generator of the GAN.

FC-DenseNet outperforms all other semantic segmentation neural networks in
numerical accuracy and visual results. On one hand, DenseNet block
concatenates different features learned by convolution layers, which can boost
the input diversity of subsequent layers and promote better efficiency of the
training. On the other hand, the detailed spatial information can be
propagated by shortcut connections between the convolution and deconvolution
paths, which enhances the recovery of fine-grained segmentation from the
deconvolution path.

####  4.4.2 Proposed framework with different DSFE

In order to choose the best feature extractor for our task, three
representative DCNNs have been adopted in the proposed framework with the
graph convolutional network. The statistical result is shown in Table 2.

Methods | OA | F1 | IoU
---|---|---|---
DSFE(U-Net)-GCN | 0.8396 | 0.6258 | 0.4544
DSFE(FCN-8s)-GCN | 0.8594 | 0.6320 | 0.4611
DSFE(FC-DenseNet)-GCN | 0.8640 | 0.6677 | 0.5012
Table 2: Quantitative comparison of different deep neural networks on
Planetscope’s datasets

From Table 2 we can see that different DCNNs exhibit different capabilities
for feature embedding. It is clear that FC-DenseNet, as a feature extractor in
DSFE with GCN, produces the best result. This is due to the superiority of FC-
DenseNet, which extends the DenseNet architecture to a U-Net-like network for
semantic segmentation. In the DenseNet block, through feature reuse, there are
shorter connections between layers close to the input and those close to the
output, which force the intermediate layers to learn discriminative features.
Moreover, DenseNet combines features by iteratively concatenating them, which
contributes to improved information and gradient propagation in the networks.

(a)

(b)

(c)

(d)

Figure 5: Visualized comparison of the predicted results using different DCNNs
in DSFE with GCN. The yellow bounding box highlights the key region for
comparison. (a) DSFE (U-Net)-GCN; (b) DSFE (FCN-8s)-GCN; (c) DSFE (FC-
DenseNet)-GCN; (d) Ground truth.

As can be seen in Fig. 5, DSFE (FC-DenseNet)-GCN gives the best result, which
implies that FC-DenseNet is a powerful tool for extracting different levels of
features.

####  4.4.3 Proposed framework with different graph models

In this section, we choose FC-DenseNet as the feature extractor in DSFE with
different graph models. The results are summarized in Table 3.

Methods | OA | F1 | IoU
---|---|---|---
FC-DenseNet [12] | 0.8551 | 0.6328 | 0.4628
DSFE-CRF [7] | 0.8592 | 0.6415 | 0.4757
DSFE-GCN [26] | 0.8640 | 0.6677 | 0.5012
DSFE-GraphSAGE [27] | 0.8719 | 0.6726 | 0.5067
DSFE-GGNN [28] | 0.8787 | 0.6778 | 0.5123
DSFE-GGCN | 0.8881 | 0.6899 | 0.5251
Table 3: Comparison of different networks on the Planetscope dataset

The results show that DSFE-GGCN has the best performance for our task. The IoU
increases 6.2% compared to the best result of DCNN. Fig. 6 shows a visual
comparison of all the networks used in section 4. We marked the key region
with a yellow bounding box. The close-up figures for the key regions are shown
in Fig. 7.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 6: Visualized comparison of the predicted results using different
networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-DUC; (d) E-Net; (e) SegNet; (f)
U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-DenseNet; (j) DSFE-CRF; (k) DSFE-GCN;
(l) DSFE-GraphSAGE; (m) DSFE-GGNN; (n) DSFE-GGCN; (o) Ground truth; (p)
Optical image.

|  |  |
---|---|---|---
|  |  |
|  |  |
|  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 7: Visualized comparison of the predicted results using different
networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-DUC; (d) E-Net; (e) SegNet; (f)
U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-DenseNet; (j) DSFE-CRF; (k) DSFE-GCN;
(l) DSFE-GraphSAGE; (m) DSFE-GGNN; (n) DSFE-GGCN; (o) Ground truth; (p)
Optical image.

##  5 Discussion

###  5.1 Additional dataset

We validate our proposed method with experiments on the ISPRS 2D Semantic
Labeling Contest dataset, which covers the city of Potsdam and comprises 38
tiles of aerial imagery [32]. In order to maintain the consistency, images
with 3 spectral bands (red, green, blue) are used in this experiment without a
digital surface model (DSM). Each aerial image is depicted with $6000\times
6000$ pixels at a spatial resolution of 5 cm. The corresponding ground truth
is also provided for results evaluation, which includes six classes:
Impervious surfaces, Building, Low vegetation, Trees, Cars, and
Clutter/background. For our detailed experiments, we split those 38 tiles into
a training subset (tile numbers 2-10 to 6-15) and a test subset (tile numbers
7-07 to 7-13). The building class is regarded as a building and other five
classes are considered non-buildings. We cut 16,000 patches of $256\times 256$
pixels from the training subset and 3573 patches from the test subset. As
mentioned in the previous section, the data augmentation step TSDM is used for
the medium-resolution images and the ground truth is well coregistrated with
the optical image. Therefore, there is no data preprocessing step for the
ISPRS dataset. The optical image is fed directly into the networks.

###  5.2 Experimental setup

The SGD optimizer is adopted and the initial learning rate is set to be 10e-4,
which is reduced by a factor of ten when the validation loss is saturated.
Once the learning rate is reduced below 10e-8, the training stops. The number
of epochs is in the range (120, 160) for all the networks. The size of the
training batch is 4.

###  5.3 Experimental results

The metrics OA, F1 scores, and IoU scores are used to evaluate the results.
Fig. 8 shows the visualized comparison of the predicted results the ISPRS
Potsdam dataset, using different networks.

FCN-8s provides a significantly higher percentage of buildings detected
compared to FCN-16s and FCN-32s, by combining predictions from not only the
final layer but also coarse layers, allowing more information to be preserved.
The boundaries of buildings detected from U-Net are sharper than for SegNet or
E-Net. However, unlike in the medium-resolution case, the completeness of the
result obtained by SegNet or E-Net is better than for U-Net, which indicates
that the spatial information propagation is more effectively undertaken by
recording the pooling indices than by concatenating the low-level features
when the resolution is high enough, i.e., when comprehensive spatial
information exists. The finer details are captured by the proposed framework
with different graph models such as CRFasRNN, GCN, and GGCN rather than CNN-
only methods, which confirms the effectiveness of the graph model in modelling
the interaction among pixels and spatial information propagation. Compared to
CRFasRNN and GCN, the proposed GGCN method gives a better result. A close-up
view of the key region is shown in Fig. 9. It can be seen that the DSFE-GGCN
shows a better result with respect to both completeness and sharpness for
building extraction compared to other methods.

|
---|---
|
|

(a)

(b)

(c)

(d)

(e)

(f)

Figure 8: Visualized comparison of the predicted results from the ISPRS
Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-
DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-
DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n)
Optical image.

|
---|---
|
|

(g)

(h)

(i)

(j)

(k)

(l)

Figure 8: Visualized comparison of the predicted results from the ISPRS
Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-
DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-
DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n)
Optical image.

|  |
---|---|---

(m)

(n)

Figure 8: Visualized comparison of the predicted results from the ISPRS
Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-
DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-
DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n)
Optical image.

|
---|---
|
|

(a)

(b)

(c)

(d)

(e)

(f)

Figure 9: Visualized comparison of the predicted results on the ROI of ISPRS
Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-
DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-
DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n)
Optical image.

|
---|---
|
|

(g)

(h)

(i)

(j)

(k)

(l)

Figure 9: Visualized comparison of the predicted results on the ROI of ISPRS
Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-
DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-
DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n)
Optical image.

|  |
---|---|---

(m)

(n)

Figure 9: Visualized comparison of the predicted results on the ROI of ISPRS Potsdam dataset using different networks. (a) FCN-32; (b) FCN-16s; (c) ResNet-DUC; (d) E-Net; (e) SegNet; (f) U-Net; (g) FCN-8s; (h) CWGAN-GP; (i) FC-DenseNet; (j) DSFE-CRF; (k) DSFE-GCN; (l) DSFE-GGCN; (m) Ground truth; (n) Optical image. Methods | OA | F1 | IoU
---|---|---|---
FCN-32s | 0.7371 | 0.6186 | 0.4478
FCN-16s | 0.8247 | 0.7429 | 0.5910
ResNet-DUC | 0.7475 | 0.6766 | 0.5051
E-Net | 0.7711 | 0.7764 | 0.6110
SegNet | 0.8948 | 0.8511 | 0.7408
U-Net | 0.8892 | 0.8392 | 0.7229
FCN-8s | 0.8617 | 0.7986 | 0.6647
CWGAN-GP | 0.8926 | 0.8504 | 0.7397
FC-DenseNet | 0.9186 | 0.9182 | 0.8789
DSFE-GCN | 0.9221 | 0.9375 | 0.9097
DSFE-GGCN | 0.9271 | 0.9422 | 0.9196
Table 4: Comparison between different deep convolutional neural networks and
proposed framework on the ISPRS dataset

Table 4 summarizes the results of using different deep convolutional neural
networks and the proposed framework on the ISPRS dataset. As can be seen the
proposed DSFE-GGCN/DSFE-GCN framework contributes a significant improvement
over the DCNNs. Moreover, compared to DSFE-GCN, DSFE-GGCN can effectively
propagate the information in the short- and long-range, which leads to better
results.

##  6 Conclusion

In this work, we develop a novel framework for semantic segmentation that
combines the deep structured feature embedding and a graph convolutional
network. Specifically, we propose using a gated graph convolutional network to
improve the information propagation by using RNN with GCN. Our proposed
framework outperforms the state-of-the-art methods for building footprint
extraction. Although we have used building footprint extraction as the
practical application, the proposed method can be generally applied to other
binary or multi-label segmentation tasks, such as road extraction, settlement
layer extraction, or semantic segmentation of very high resolution data in
general. In addition, the proposed GCN network can work directly with
unstructured data, such as point clouds and social media text messages.

##  7 Acknowledgments

This work is supported by the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation programme (grant
agreement no. ERC-2016-StG-714087, acronym: So2Sat, www.so2sat.eu), the
Helmholtz Association under the framework of the Young Investigators Group
“SiPEO” (VH-NG-1018, www.sipeo.bgu.tum.de), Munich Aerospace e.V. Fakultät für
Luft- und Raumfahrt, and the Bavaria California Technology Center (Project:
Large-Scale Problems in Earth Observation). The authors thank the Gauss Centre
for Supercomputing (GCS) e.V. for funding this project by providing computing
time on the GCS Supercomputer SuperMUC at the Leibniz Supercomputing Centre
(LRZ) and on the supercomputer JURECA at Forschungszentrum Jülich. The authors
thank Planet for providing the datasets.

##  8 References

## References

  * [1] A. O. Ok, “Automated detection of buildings from single VHR multispectral images using shadow information and graph cuts,” _ISPRS J. Photogramm. Remote Sens._ , vol. 86, pp. 21-40, 2013.
  * [2] Y. Xu, L. Wu, Z. Xie, and Z. Chen, “Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters,” _Remote Sensing_ , vol. 10, pp. 144, Jan. 2018.
  * [3] K. Bittner, F. Adam, S. Cui, M. Körner and P. Reinartz, “Building Footprint Extraction From VHR Remote Sensing Images Combined With Normalized DSMs Using Fused Fully Convolutional Networks,” _IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens._ , vol. 11, pp. 2615-2629, 2018.
  * [4] Q. Chen, L. Wang, Y. Wu, G. Wu, Z. Guo, and S. Waslander, “Aerial imagery for roof segmentation: A large-scale dataset towards automatic mapping of buildings,”, _ISPRS J. Photogramm. Remote Sens._ , vol. 147, pp. 42-55, 2019.
  * [5] X. X. Zhu, D. Tuia, L. Mou, G. Xia, L. Zhang, F. Xu, and F. Fraundorfer, “Deep learning in remote sensing: a comprehensive review and list of resources,” _IEEE Geosci. Remote Mag._ , vol. 5, no. 4, pp. 8-36, 2017.
  * [6] Z. Liu, X. Li, P. Luo, C. C. Loy, X. Tang, “Semantic Image Segmentation via Deep Parsing Network,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 1377-1385.
  * [7] L. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 4, pp. 834-848, 2017.
  * [8] J. Long, E. Shelhamer, T. Darrell, “Fully Convolutional Networks for Semantic Segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 3431-3440.
  * [9] V. Badrinarayanan, A. Handa, R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling,” _arXiv preprint arXiv:1505.07293_ , 2015.
  * [10] O. Ronneberger, P. Fischer, T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Proc. Int. Conf. Med. Image Comput. Comput.-Assisted Intervention_ , 2015, pp. 234-241.
  * [11] S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, P. Torr, “Conditional random fields as recurrent neural networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 1529-1537.
  * [12] S. Jégou, M. Drozdzal, D. Vázquez, A. Romero, Y. Bengio, “The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation,” _arXiv preprint arXiv:1611.09326_ , 2017.
  * [13] G. Huang, Z. Liu, K. Q. Weinberger, L. van der Maaten, “Densely connected convolutional networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 4700-4708.
  * [14] T. Akilan, Q. M. Wu, W. Jiang, “A feature embedding strategy for high-level CNN representations from multiple convnets,” in _Proc. IEEE Conf. Signal and Information Processing_ , 2017, pp. 1195-1199.
  * [15] H. Noh, S. Hong, B. Han, “Learning Deconvolution Network for Semantic Segmentation,” in _Proc. Int. Conf. Comput. Vision_ , 2015, pp. 1520-1528.
  * [16] O. A. Penatti, K. Nogueira, J. A. dos Santos, “Do Deep Features Generalize from Everyday Objects to Remote Sensing and Aerial Scenes Domains?” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops_ , 2015, pp. 44-51.
  * [17] J. Yuan, “ Learning Building Extraction in Aerial Scenes with Convolutional Networks,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 11, pp. 2793-2798, Nov. 2018.
  * [18] D. Marcos, D. Tuia, B. Kellenberger, L. Zhang, M. Bai, R. Liao, and R. Urtasun, “Learning deep structured active contours end-to-end,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018.
  * [19] J. Huang, X. Zhang, Q. Xin, Y. Sun, and P. Zhang, “Automatic building extraction from high-resolution aerial images and LiDAR data using gated residual refinement network,”, _ISPRS J. Photogramm. Remote Sens._ , vol. 151, pp. 91-105, 2019.
  * [20] Y. Shi, Q. Li, X. X. Zhu, “Building Footprint Generation Using Improved Generative Adversarial Networks,” _IEEE Geosci. Remote Lett._ , 2018, doi:10.1109/LGRS.2018.2878486.
  * [21] S. Wang, M. Bai, G. Mattyus, H. Chu, W. Luo, B. Yang, J. Liang, J. Cheverie, S. Fidler, and R. Urtasun. “TorontoCity: Seeing the world with a million eyes,” in _Proc. Int. Conf. Comput. Vision_ , 2017.
  * [22] S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, S. Lin, “Graph embedding and extensions: A general framework for dimensionality reduction,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 29, no. 1, pp. 40-51, Jan. 2007.
  * [23] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun, “Spectral networks and locally connected networks on graphs,” _arXiv preprint arXiv:1312.6203_ , 2013.
  * [24] M. Henaff, J. Bruna, and Y. LeCun, “Deep convolutional networks on graph-structured data,” _arXiv preprint arXiv:1506.05163_ , 2015.
  * [25] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in _Proc. Int. Conf. Neural Information Processing Systems (NIPS)_ , 2016, pp. 3844-3852.
  * [26] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [27] W.L. Hamilton, R. Ying, and J. Leskovec, “Inductive Representation Learning on Large Graphs,” _arXiv preprint arXiv:1706.02216_ , 2017.
  * [28] Y. Li, D. Tarlow, M. Brockschmidt, R. Zemel, “Gated Graph Sequence Neural Networks,” in _Proc. Int. Conf. Learning Representations_ , 2016.
  * [29] K. Cho, B. Merrie, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk and Y. Bengio, “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation,” _arXiv preprint arXiv:1406.1078v3_ , 2014.
  * [30] PlanetScope. https://www.planet.com/
  * [31] Openstreetmap. http://www.openstreetmap.org
  * [32] ISPRS 2D Semantic Labeling Dataset - Potsdam. http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-potsdam.html