# Complete ADHM Sigma Model

Abbas Ali and P.P. Abdul Salih
Physics Department, Aligarh Muslim University,
Aligarh, India

###### Abstract

The field theoretic ADHM instanton has a stringy generalization as a linear
sigma model constructed by Witten in 1995. Recently we constructed a
complementary version related to it by a duality. In this note we generalize
these to a complete ADHM instanton linear sigma model in which above duality
is promoted to a $Z_{2}$ symmetry.

##  1 Introduction

The SU(2) Yang-Mills instanton by ’t Hooft is a field theoretic construct that
depends upon $5k+4$ parameters for topological charge $k$. This can be
promoted to string theory using sigma model approach. Corresponding worldsheet
construction was done in Ref.[1] by Callan, Harvey and Strominger. It obeys
the low enegy effective equations derived from heterotic strings.
Corresponding solution possesses (0,4) supersymmetry. There is a worldbrane
version too that was studied in Ref.[2] while overall structure was reviewed
in Ref.[3].

The full SU(2) Yang-Mills instanton can have $8k$ parameters for topological
charge $k$ and this was constructed by Atiyah, Drinfeld, Hitchin and Manin in
1978 in a remarkably brief and linear algebraic construction in Ref.[4]. This
construction used inspirations from twistor theory [5]. In Ref.[6] it was
explained in terms of the differential geometry of $R^{4}$. Further accessible
version and generalization to other groups were presented in Ref.[7]. A recent
mathematical exposition of the construction is covered in Ref.[8]. Edward
Witten gave us an equally remarkable stringy generalization using linear, that
is massive, sigma model in Ref.[9]. Insights related to renormalization and
quantization of this ADHM instanton sigma model including some clarifcation of
superfield structure were presented in Ref.[10] by Lambert. The version to
include D-branes was constructed by Douglas in Ref.[11]. Related p-brane and
D-brane aspects were further investigated in Refs.[12, 13]. The aspects
related to the geometry of the semi-infinite throat [3] were investigated by
Johnson in Ref.[14].

In spite of these deep and powerful insights the ADHM sigma model is not as
well explored and exploited as the simpler version of the Callan, Harvey and
Strominger for the ’t Hooft case. This situation should change with the
results of Ref.[15] where an ADHM sigma model complementary to the one in
Ref.[9] was constructed by us (see also [16, 17]. The two models, Witten’s
original and our complementary, are related to each other by a simple duality.
A version of this duality was already perceptible in Ref.[13]. The
construction in Ref. [15] resolves the mystery that was recorded in [9] and
was further analyzed in Ref.[18] without mitigation. Its quantization has been
discussed by us in Ref.[19]. It has already helped us in resolving the long
standing problem of doubling of the Ramond superalgebra encountered in the
well known study of $AdS_{3}$ superstrings in Ref.[20]. The present note
contains the construction of a complete ADHM sigma model as well as discussion
of many additional complications. It is complete in the sense that this
construction incorporates the duality between the original and complementary
model as a $Z_{2}$ symmetry.

The rest of this article is organized as follows. In Section 2 we spell out
the notation, set out the background, collect ingredients from the original
and complementary ADHM instanton sigma models and combine these to synthesize
the complete model. It has a $(k,k^{\prime})$ instanton structure, that is, it
possesses two topological charges. The $(1,1)$ instanton case,
$(k=1,k^{\prime}=1)$, is taken up in Section 3 to discuss the structure of
resulting moduli space. We discuss the other aspects of the resulting model in
Section 4.

##  2 Notation, Background, Ingredients, Synthesis

We begin by collecting in this section those essential ingredients of the
original ADHM linear sigma model that will be part of the final model. These
can be introduced either following the lines in the original approach of
Ref.[9] or the one adopted in [10]. In Ref. [15] we used the latter while in
the present we shall employ the former.

We begin with bosons $X^{AY}$ and $\phi^{A^{\prime}Y^{\prime}}$ with $A$,
$A^{\prime}$=1,2, $Y=1,2,...,2k$ and $Y^{\prime}=1,2,...,2k^{\prime}$. The $A$
and $A^{\prime}$ indices are raised (lowered) with the antisymmetric $Sp(1)$
tensors $\epsilon^{AB}$($\epsilon_{AB}$) and
$\epsilon^{A^{\prime}B^{\prime}}$($\epsilon_{A^{\prime}B^{\prime}}$)
respectively. Similarly the $Y$ and $Y^{\prime}$ indices are raised(lowered)
with the help of the antisymmetric $Sp(k)$ and $Sp(k^{\prime})$ tensors
$\epsilon^{YZ}$($\epsilon_{YZ}$) and
$\epsilon^{Y^{\prime}Z^{\prime}}$($\epsilon_{Y^{\prime}Z^{\prime}}$)
respectively. The light cone coordinates $\sigma^{\pm}$ are defined as
$\sigma^{\pm}=(\tau\pm\sigma)/\sqrt{2}$ and
$\partial_{\pm}=\frac{1}{\sqrt{2}}(\partial_{0}\pm\partial_{1})$ while the
worldsheet metric is taken as $ds^{2}=d\tau^{2}-d\sigma^{2}$. The fermionic
partners of $X$ and $\phi$ bosons are $\psi_{-}^{A^{\prime}Y}$ and
$\chi_{-}^{AY^{\prime}}$ respectively and corresponding supersymmetry
transformations are

$\displaystyle\delta_{\eta}X^{AY}$ $\displaystyle=$ $\displaystyle
i\epsilon_{A^{\prime}B^{\prime}}\eta^{AA^{\prime}}_{+}\psi^{B^{\prime}Y}_{-},~{}~{}\delta_{\eta}\psi^{A^{\prime}Y}_{-}=\epsilon_{AB}\eta^{AA^{\prime}}_{+}\partial_{-}X^{BY},$
$\displaystyle\delta_{\eta}\phi^{A^{\prime}Y^{\prime}}$ $\displaystyle=$
$\displaystyle
i\epsilon_{AB}\eta^{AA^{\prime}}_{+}{\chi}^{BY^{\prime}}_{-},~{}~{}~{}\delta_{\eta}{\chi}^{AY^{\prime}}_{-}=\epsilon_{A^{\prime}B^{\prime}}\eta^{AA^{\prime}}_{+}\partial_{-}\phi^{B^{\prime}Y^{\prime}}.$
(1)

There are supercharges $Q^{AA^{\prime}}$ that obey the superalgebra

$\\{Q^{AA^{\prime}},Q^{BB^{\prime}}\\}=\epsilon^{AB}\epsilon^{A^{\prime}B^{\prime}}P^{+},~{}~{}~{}P^{+}=P_{-}=-i\partial/\partial\sigma^{-}.$
(2)

The fields and supercharges have the following reality conditions

$X^{AY}=\epsilon^{AB}\epsilon^{YZ}\overline{X}_{BZ},~{}~{}~{}\phi^{A^{\prime}Y^{\prime}}=\epsilon^{A^{\prime}B^{\prime}}\epsilon^{Y^{\prime}Z^{\prime}}\overline{\phi}_{B^{\prime}Z^{\prime}},~{}~{}~{}Q^{AA^{\prime}}=\epsilon^{AB}\epsilon^{A^{\prime}B^{\prime}}Q^{\dagger}_{BB^{\prime}}.$
(3)

Additionally there are left moving fermions $\lambda_{+}^{a}$, $a=1,\dots,n$
with the action

${i\over 2}\int
d^{2}\sigma\left(\lambda_{+}^{a}\partial_{-}\lambda_{+}^{a}-\lambda_{+}^{a}G_{a\theta}\rho_{-}^{\theta}\right),$
(4)

where the factor $\rho_{-}^{\theta}$ in the Yukawa term contains both $\psi$’s
and $\chi$’s. The expression for $G_{a\theta}$ is fixed using the equation of
motion of $\lambda_{+}^{a}$

$\partial_{-}\lambda_{+}^{a}=G^{a}_{\theta}\rho_{-}^{\theta}$ (5)

such that the most general supersymmetry transformation of $\lambda_{+}^{a}$

$\delta\lambda_{+}^{a}=\eta_{+}^{AA^{\prime}}C_{AA^{\prime}}^{a},$ (6)

is in consonance with supersymmetry algebra (2) above. Eqn.(6) gives

$\displaystyle\delta_{\eta^{\prime}}\delta_{\eta}\lambda_{+}^{a}=i\eta_{+}^{AA^{\prime}}\left({\partial
C_{AA^{\prime}}^{a}\over\partial
X^{BY}}\epsilon_{B^{\prime}C^{\prime}}{\eta^{\prime}}_{+}^{BB^{\prime}}\psi_{-}^{C^{\prime}Y}+{\partial
C_{AA^{\prime}}^{a}\over\partial\phi^{B^{\prime}Y^{\prime}}}\epsilon_{BC}{\eta^{\prime}}_{+}^{BB^{\prime}}\chi^{CY^{\prime}}_{-}\right),$
(7)

such that

$\displaystyle[\delta_{\eta^{\prime}},\delta_{\eta}]=-i\epsilon_{AB}\epsilon_{A^{\prime}B^{\prime}}\eta_{+}^{AA^{\prime}}\eta^{\prime}{}_{+}^{BB^{\prime}}\partial_{-}\lambda_{+}^{a}$
(8)
$\displaystyle=-i\epsilon_{AB}\epsilon_{A^{\prime}B^{\prime}}\eta_{+}^{AA^{\prime}}\eta^{\prime}{}_{+}^{BB^{\prime}}G^{a}_{\theta}\rho^{\theta}.$
(9)

Eqn.(7) conforms with (9) provided

$\displaystyle{\partial C_{AA^{\prime}}^{a}\over\partial X^{BY}}+{\partial
C_{BA^{\prime}}^{a}\over\partial X^{AY}}={\partial
C_{AA^{\prime}}^{a}\over\partial\phi^{B^{\prime}Y^{\prime}}}+{\partial
C_{AB^{\prime}}^{a}\over\partial\phi^{A^{\prime}Y^{\prime}}}=0.$ (10)

To obtain the Lagrangian with above symmetry $\lambda^{a}$ is taken to be a
part of a supermultiplet with

$\Lambda^{a}=\lambda^{a}+\theta F^{a},~{}~{}~{}\delta\lambda^{a}=\eta F^{a}$
(11)

such that potential becomes

$V_{W}={1\over 2}\sum_{a}F^{a}F^{a}={1\over 2}\sum_{a}C^{a}C^{a}$ (12)

with the definition and condition where $C^{a}$ are defined using $c$-numbers
$c^{AA^{\prime}}$ with following normalization condition

$C^{a}=c^{AA^{\prime}}C_{AA^{\prime}}^{a},~{}~{}~{}\epsilon_{AB}\epsilon_{A^{\prime}B^{\prime}}c^{AA^{\prime}}c^{BB^{\prime}}=1.$
(13)

The condition for the Lagrangian to be invariant under all supersymmetries is

$\sum_{a}\left(C_{AA^{\prime}}^{a}C_{BB^{\prime}}^{a}+C_{BA^{\prime}}^{a}C_{AB^{\prime}}^{a}\right)=0.$
(14)

These are the ingredients from the original model. Then there are
corresponding ingredients from the complementary sector with
$\lambda^{a}\rightarrow\hat{\lambda}^{\hat{a}}$, $\hat{a}=1,...,n^{\prime}$,
$G^{a}_{\theta}\rightarrow\hat{G}^{\hat{a}}_{\theta}$,
$C^{a}_{AA^{\prime}}\rightarrow\hat{C}^{\hat{a}}_{AA^{\prime}}$,
$\Lambda^{a}\rightarrow\hat{\Lambda}^{\hat{a}}$,
$F^{a}\rightarrow\hat{F}^{\hat{a}}$ and
$c^{AA^{\prime}}\rightarrow\hat{c}^{AA^{\prime}}$ with corresponding equations
duplicating Eqs.(4-14). The full Lagrangian is obtained by judicious
combination of elements from original and complementary models. It is

$L=L_{kin}+L_{W}+L_{C}$ (15)

where $L_{W}$ is

$\displaystyle L_{W}=-{i\over 4}\int
d^{2}\sigma\lambda_{+}^{a}\left(\epsilon^{BD}{\partial
C^{a}_{BB^{\prime}}\over\partial
X^{DY}}\psi_{-}^{B^{\prime}Y}+\epsilon^{B^{\prime}D^{\prime}}{\partial
C^{a}_{BB^{\prime}}\over\partial\phi^{D^{\prime}Y^{\prime}}}\chi_{-}^{BY^{\prime}}\right)$
(16) $\displaystyle-{1\over 8}\int
d^{2}\sigma\epsilon^{AB}\epsilon^{A^{\prime}B^{\prime}}C^{a}_{AA^{\prime}}C^{a}_{BB^{\prime}}$
(17)

with $C^{a}_{AA^{\prime}}$ given by Eqn.(21).

$\displaystyle L_{C}=-{i\over 4}\int
d^{2}\sigma\hat{\lambda}_{+}^{a}\left(\epsilon^{BD}{\partial\hat{C}^{a}_{BB^{\prime}}\over\partial
X^{DY}}\psi_{-}^{B^{\prime}Y}+\epsilon^{B^{\prime}D^{\prime}}{\partial\hat{C}^{a}_{BB^{\prime}}\over\partial\phi^{D^{\prime}Y^{\prime}}}\chi_{-}^{BY^{\prime}}\right)$
(18) $\displaystyle-{1\over 8}\int
d^{2}\sigma\epsilon^{AB}\epsilon^{A^{\prime}B^{\prime}}\hat{C}^{a}_{AA^{\prime}}\hat{C}^{a}_{BB^{\prime}}.$
(19)

To obtain explicit form of $C_{AA^{\prime}}^{a}$ we begin with the following
general form that is separately linear in both $\phi$ and $X$.

$\displaystyle C_{AA^{\prime}}^{a}$
$\displaystyle=M_{AA^{\prime}}^{a}+X_{AY}N^{a\,Y}_{A^{\prime}}+\phi_{A^{\prime}}{}^{Y^{\prime}}D_{AY^{\prime}}^{a}+X_{A}{}^{Y}\phi_{A^{\prime}}{}^{Y^{\prime}}E^{a}_{YY^{\prime}}.$
(20)

In Witten’s orginal model the choice is to take $M=N=0$ such that

$\displaystyle{C}^{a}_{AA^{\prime}}=\epsilon_{A^{\prime}B^{\prime}}D^{a}_{AY^{\prime}}\phi^{~{}B^{\prime}Y^{\prime}}+\epsilon_{AB}\epsilon_{A^{\prime}B^{\prime}}E^{a}_{YY^{\prime}}X^{BY}\phi^{~{}B^{\prime}Y^{\prime}}\equiv\phi_{A^{\prime}}^{~{}Y^{\prime}}B^{a}_{AY^{\prime}}(X)$
(21)

such that the condition (14) reduces to

$\sum_{a}(B^{a}_{AY^{\prime}}B^{a}_{BZ^{\prime}}+B^{a}_{BY^{\prime}}B^{a}_{AZ^{\prime}})=0.$
(22)

Here $B^{a}_{AY^{\prime}}(X)$ is linear in $X$ and independent of $\phi$.

For the original model there are $N=n-4k^{\prime}$ massless components with
$v_{i}^{a},i=1,2,...,N$ and

$\sum_{a}v^{a}_{i}B^{a}_{AY^{\prime}}=0.$ (23)

The normalization condition is

$\sum_{a}v^{a}_{i}v^{a}_{j}=\delta_{ij}.$ (24)

Setting

$\lambda^{a}_{+}=\sum_{i=1}^{N}v^{a}_{i}\lambda_{+i}$ (25)

amount to putting massive modes of $\lambda^{a}_{+}$ equal to zero.

The covariant derivative and the resulting instanton field are

$\delta_{ij}\partial_{-}+A_{ijAY}\partial_{-}X^{AY},~{}~{}~{}A_{ijAY}=\sum_{a}v^{a}_{i}\frac{\partial
v^{a}_{j}}{\partial X^{AY}}$ (26)

respectively. The supersymmetry structure so far is the usual $(0,4)$ one.

Equal amount of construction is borrowed from the complementary sigma model.
The complementary model has $\hat{M}=\hat{D}=0$ in the following general form
of the tensor $\hat{C}^{\hat{a}}_{AA^{\prime}}$

$\displaystyle\hat{C}_{AA^{\prime}}^{\hat{a}}$
$\displaystyle=\hat{M}_{AA^{\prime}}^{\hat{a}}+X_{AY}\hat{N}^{\hat{a}\,Y}_{A^{\prime}}+\phi_{A^{\prime}}{}^{Y^{\prime}}\hat{D}_{AY^{\prime}}^{\hat{a}}+X_{A}{}^{Y}\phi_{A^{\prime}}{}^{Y^{\prime}}\hat{E}^{\hat{a}}_{YY^{\prime}}.$
(27)

results in

$\displaystyle\hat{C}^{\hat{a}}_{AA^{\prime}}=\epsilon_{AB}N^{\hat{a}}_{A^{\prime}Y}X^{BY}+\epsilon_{AB}\epsilon_{A^{\prime}B^{\prime}}E^{\hat{a}}_{YY^{\prime}}X^{BY}\phi^{~{}B^{\prime}Y^{\prime}}\equiv
X_{A}^{~{}Y}A^{\hat{a}}_{A^{\prime}Y}(\phi).$ (28)

The condition corresponding to Eqn.(14)

$\sum_{\hat{a}}\left(\hat{C}_{AA^{\prime}}^{\hat{a}}\hat{C}_{BB^{\prime}}^{\hat{a}}+\hat{C}_{BA^{\prime}}^{\hat{a}}\hat{C}_{AB^{\prime}}^{\hat{a}}\right)=0$
(29)

which becomes

$\sum_{\hat{a}}(A^{\hat{a}}_{A^{\prime}Y}A^{\hat{a}}_{B^{\prime}Z}+A^{\hat{a}}_{B^{\prime}Y}A^{\hat{a}}_{A^{\prime}Z})=0.$
(30)

Here $A^{\hat{a}}_{A^{\prime}Y}(\phi)$ is linear in $\phi$ and independent of
$X$.

For complementary model there are $N^{\prime}=n^{\prime}-4k$ massless
components $\hat{v}_{\hat{i}}^{\hat{a}},\hat{i}=1,2,...,N^{\prime}$ with

$\sum_{\hat{a}}\hat{v}^{\hat{a}}_{\hat{i}}A^{\hat{a}}_{A^{\prime}Y}=0.$ (31)

For the complementary part the normalization condition will be

$\sum_{\hat{a}}\hat{v}^{\hat{a}}_{\hat{i}}\hat{v}^{\hat{a}}_{\hat{j}}=\delta_{\hat{i}\hat{j}}.$
(32)

Eqn. corresponding to (25) will be

$\hat{\lambda}^{\hat{a}}_{+}=\sum_{\hat{i}=1}^{N^{\prime}}\hat{v}^{\hat{a}}_{\hat{i}}\hat{\lambda}_{+\hat{i}}$
(33)

that amounts to putting the massive fermions to zero.

In the complementary case the covariant derivative has the expression

$\delta_{\hat{i}\hat{j}}\partial_{-}+A_{\hat{i}\hat{j}A^{\prime}Y^{\prime}}\partial_{-}\phi^{A^{\prime}Y^{\prime}}$
(34)

with the instanton

$A_{\hat{i}\hat{j}A^{\prime}Y^{\prime}}=\sum_{\hat{a}}\hat{v}^{\hat{a}}_{\hat{i}}\frac{\partial\hat{v}^{\hat{a}}_{\hat{j}}}{\partial\phi^{A^{\prime}Y^{\prime}}}.$
(35)

This completes our construction of the complete ADHM instanton sigma model by
amalgamation of Witten’s original and our complementary model. The resulting
model has a double, that is, a $(k,k^{\prime})$ instanton structure means that
it has two indepedent instanton charges. With these additional Yukawa
couplings the supersymmetry structure should be a generalization of the
$(0,4)$ supersymmetry and we would term it as large $(0,4)$. Here we are
anticipating this new supersymmetry structure with analogy to the
superconformal case. In our knowledge corresponding algebra has not been
worked out.

Let us now analyze how the vacuum given by the two branches.
$C^{a}_{AA^{\prime}}$ is homogeneous and linear in $\phi$ and the resulting
potential $V_{W}$ (for $k=k^{\prime}=1$ expression see next section) is
homogeneous and quadratic in $\phi$. For $\phi=0$ it vanishes for any $X$.
This implies that $X$’s are massless fields and $\phi$’s are massive. The
potential is interpreted as a $X$ dependant mass term for the $\phi$’s. For
large enough number $n$ of $\lambda$’s the solution for $B^{a}_{AA^{\prime}}$
tensor is generic and for every value of $X$ all of the $\phi$’s are massive.
Thus the moduli space, that is the space of vacua $V_{W}$ is parametrized by
the $X$’s and it is

$\mathcal{M}_{W}={R}^{4k}.$ (36)

Here $W$ stands for Witten’s original construction and this analysis is valid
for vanishing instanton ($\omega=0$) of the complementary sector. The
complementary branch gives the moduli space factor

$\mathcal{M}_{C}={R}^{4k^{\prime}}.$ (37)

The details of the argument are in parallel to the details above for the
original branch. The total moduli space, therefore, is

$\mathcal{M}=\mathcal{M}_{W}\otimes\mathcal{M}_{C}={R}^{4k}\otimes{R}^{4k^{\prime}}.$
(38)

It has $Z_{2}$ symmetry $k\leftrightarrow k^{\prime}$.

##  3 The (1,1) Instanton Case

To analyze the structure of the the moduli space in case of the original and
the complementary models it is convenient to go to $(1,1)$ instanton cases.
Here the single instantons from both original and complementary branches are
present.

When we put $k=1$ in the former case and $k^{\prime}=1$ in the latter case to
get $R^{4}$ spacetime from both cases we realize that the minimal non-trivial
construction lives on $R^{8}$. The resulting models have $F^{\prime}\times
K\times SU(2)_{L}\times SU(2)_{R}\cong SU(2)^{4}$ invariance in the original
model sector and $F\times K^{\prime}\times SU(2)^{\prime}_{L}\times
SU(2)^{\prime}_{R}\cong SU(2)^{4}$ in the complementary model sector.

The resulting left moving massless fermions in the original model are
$\lambda_{+}^{AY^{\prime}}$ and $\lambda_{+}^{YY^{\prime}}$ with the reality
conditions

$\displaystyle\lambda^{AY^{\prime}}_{+}=\epsilon^{AB}\epsilon^{Y^{\prime}Z^{\prime}}\overline{\lambda}_{+BZ^{\prime}},~{}$
$\displaystyle\lambda^{YY^{\prime}}_{+}=\epsilon^{YZ}\epsilon^{Y^{\prime}Z^{\prime}}\overline{\lambda}_{+ZZ^{\prime}}$
(39)

while in the complementary case these are $\hat{\lambda}_{+}^{A^{\prime}Y}$
and $\hat{\lambda}_{+}^{YY^{\prime}}$ with reality conditions

$\displaystyle\hat{\lambda}^{A^{\prime}Y}_{+}=\epsilon^{A^{\prime}B^{\prime}}\epsilon^{YZ}\overline{\hat{\lambda}}_{+B^{\prime}Z},~{}$
$\displaystyle\hat{\lambda}^{YY^{\prime}}_{+}=\epsilon^{YZ}\epsilon^{Y^{\prime}Z^{\prime}}\overline{\hat{\lambda}}_{+ZZ^{\prime}}.$
(40)

The Yukawa couplings of the original model are

$\displaystyle{C}^{YY^{\prime}}_{~{}~{}~{}~{}BB^{\prime}}=X_{B}^{~{}Y}\phi^{~{}Y^{\prime}}_{B^{\prime}},~{}$
$\displaystyle{C}^{AY^{\prime}}_{~{}~{}~{}~{}BB^{\prime}}=\frac{\rho}{\sqrt{2}}\delta^{A}_{~{}B}\phi^{~{}Y^{\prime}}_{B^{\prime}}$
(41)

and of the complementary model are as follows

$\displaystyle\hat{C}^{YY^{\prime}}_{~{}~{}~{}~{}BB^{\prime}}=X_{B}^{~{}Y}\phi^{~{}Y^{\prime}}_{B^{\prime}},~{}$
$\displaystyle\hat{C}^{A^{\prime}Y}_{~{}~{}~{}~{}BB^{\prime}}=\frac{\omega}{\sqrt{2}}\delta^{A^{\prime}}_{~{}B^{\prime}}X^{~{}Y}_{B}.$
(42)

The resulting potentials for Witten’s original and our complementary model are

$V_{W}=\frac{1}{8}(X^{2}+\rho^{2})\phi^{2},~{}~{}~{}V_{C}=\frac{1}{8}(\phi^{2}+\omega^{2})X^{2}$
(43)

respectively. Using the fermionic field ansätz

$\displaystyle\lambda^{YY^{\prime}}=\frac{\rho\zeta^{~{}YY^{\prime}}_{+}}{\sqrt{\rho^{2}+X^{2}}},~{}$
$\displaystyle\lambda^{AY^{\prime}}=-\frac{\sqrt{2}X^{A}_{~{}~{}Y}\zeta_{+}^{~{}YY^{\prime}}}{\sqrt{\rho^{2}+\phi^{2}}}.$
(44)

for the original case and

$\displaystyle\hat{\lambda}^{YY^{\prime}}=\frac{\omega\zeta^{~{}YY^{\prime}}_{+}}{\sqrt{\omega^{2}+\phi^{2}}},~{}$
$\displaystyle\hat{\lambda}^{A^{\prime}Y}=-\frac{\sqrt{2}\phi^{A^{\prime}}_{~{}~{}Y^{\prime}}\zeta_{+}^{~{}YY^{\prime}}}{\sqrt{\omega^{2}+\phi^{2}}}$
(45)

for the complementary case, the instanton expressions take the forms

$\displaystyle\lambda_{+YY^{\prime}}\partial_{-}\lambda^{YY^{\prime}}_{+}+\lambda_{+AY^{\prime}}\partial_{-}\lambda^{AY^{\prime}}_{+}=\zeta_{+YY^{\prime}}\partial_{-}\zeta^{~{}YY^{\prime}}_{+}-\zeta_{+YY^{\prime}}$
$\displaystyle\frac{\frac{1}{2}\epsilon_{AB}(X^{AY}\partial_{-}X^{BZ}+X^{AZ}\partial_{-}X^{BY})}{X^{2}+\rho^{2}}\zeta^{~{}~{}~{}Y^{\prime}}_{+Z}$
(46)

and

$\displaystyle\hat{\lambda}_{+YY^{\prime}}\partial_{-}\hat{\lambda}^{YY^{\prime}}_{+}+\hat{\lambda}_{+A^{\prime}Y}\partial_{-}\hat{\lambda}^{A^{\prime}Y}_{+}=\hat{\zeta}_{+YY^{\prime}}\partial_{-}\hat{\zeta}^{~{}YY^{\prime}}_{+}-\hat{\zeta}_{+YY^{\prime}}$
$\displaystyle\frac{\frac{1}{2}\epsilon_{A^{\prime}B^{\prime}}(\phi^{A^{\prime}Y^{\prime}}\partial_{-}\phi^{B^{\prime}Z^{\prime}}+\phi^{A^{\prime}Z^{\prime}}\partial_{-}\phi^{B^{\prime}Y^{\prime}})}{\phi^{2}+\omega^{2}}\hat{\zeta}^{Y}_{+Z^{\prime}}$
(47)

for original and complementary branches respectively.

The total potential in case of double instanton is

$V=V_{W}+V_{C}={1\over
8}\left(2X^{2}\phi^{2}+\rho^{2}\phi^{2}+\omega^{2}X^{2}\right).$ (48)

The combined moduli space is obtained by minimizing above quadratic
expression. The structure of the resulting moduli space can be discussed in
conjuction with the two moduli spaces of the original and complementary
models.

The structures of the two moduli spaces are the following. In the original
model $\phi=0$ gives the moduli space for all values of $X$. For vanishing
instanton a new branch opens up because $V=\frac{1}{8}X^{2}\phi^{2}$ is again
zero for all values of $\phi$ for $X=0$. This was called $\phi$ branch in [9]
and it was a mystery at the time and it continued to Ref.[18].

The mystery is resolved if we analyze the moduli space for complementary
branch. Expectedly the discussion of this branch of the moduli space is
complementary.

In the complementary model $X=0$ gives the moduli space for all values of
$\phi$. In the small instanton case we again have $V=\frac{1}{8}X^{2}\phi^{2}$
and we get another branch for $\phi=0$ with all values of $X$. This would be
called the $X$ branch.

From this discussion it is clear that the extra branches are nothing but the
moduli spaces of the other model because the two models become degenerate in
the small instanton limit.

With the explanation of the origin of the additional branches the mystery is
resloved.

The structure of the moduli space of the complete model in the double
instanton case is the following. For the vanishing of one instanton, say
$\rho\rightarrow 0$, we get the $\phi$ branch for $X=0$ and any $\phi$. Then
there is another branch that appears for $\omega\rightarrow 0$, $\phi=0$ and
any $X$. The two branches are related by $X\leftrightarrow\phi$,
$\rho\leftrightarrow\omega$. As a result the dual structure of the individual
models appears as a $Z_{2}$ symmetry. This $Z_{2}$ symmetry is there for
vanishing of one or the other instanton. It persists for vanishing of both
instantons.

##  4 Discussion

The duality between the original and complementary constructions becomes a
$Z_{2}$ symmetry in the complete model. This is in consonance with what we
expect from the infrared limit of the model.

In the infrared limit, or rather the conformal limit, both original and
complementary models will have small N=4 superconformal symmetries [21, 22].
The complete model will have large N=4 superconformal symmetry [Sevrin:188ab,
24, 25, 26, 27] in the conformal limit. We need better understanding of both
of these limits.

The small N=4 superconformal symmetry has an $SU(2)$ Kac-Moody subalgebra.
This should be visible as an $SU(2)$ R symmetry before the conformal limit.
The original as well as the complementary models have a galore of $SU(2)$
symmetries before the conformal limit. The supersymmetry structure in the
present formulation is not entirely transparent. Construction of the
complementary and complete ’t Hooft instantons, taking Callan-Harvey-
Strominger as the starting original construction, should help in clarifying
these issues because the structure of supersymmetry is much more clear in that
case. These constructions should also help in clarifying the semi-infinite
throat there and its counter part here. It would also be interesting to
analyze in what limit the ADHM construction and its generalizations to higher
groups reduce to ’t Hooft construction and corresponding generalizations.

The complete model should have two $SU(2)$ R symmetries and these would be
easy to identify once we identify the $SU(2)$ R symmetries in the original and
complementary models. The large N=4 superconformal algebra has an $SU(2)\times
SU(2)\times U(1)$ Kac-Moody subalgebra and hence at least in the
superconformal limit we should be able to identify a $U(1)$ symmetry in the
complete model discussed in this note. Our speculation in this regard is the
following. In the pair of scalars $(X,\phi)$ we have two linear combinations
and one of them is $U(1)$ field while another one is a scalar field with
background charge. Clearly these aspects need further clarifications. It will
also be interesting to do Inönü-Wigner contraction of the complete model and
reach the middle N=4 superconformal symmetry and corresponding ADHM linear
sigma model [28, 25, 29].

From the resulting structure it is apparent that the quantization of the
original model discussed in Ref.[10] and the quantization of the complementary
model discussed in Ref.[19] cover all that is needed for quantization of this
model.

Acknowledgements:We thank Dr. Mohsin Ilahi for help in the preparation of the
manuscript as well as its critical reading. This work was done as part of the
Ph.D. thesis of P.P. Abdul Salih. We thank Prof. Neil D Lambert for some
clarifications and Dr. Raktim Abir for a very useful suggestion.

## References

  * [1] C. G. Callan, Jr., J. A. Harvey and A. Strominger, “World sheet approach to heterotic instantons and solitons,” Nucl. Phys. B 359 (1991) 611.
  * [2] C. G. Callan, Jr., J. A. Harvey and A. Strominger, “Worldbrane actions for string solitons,” Nucl. Phys. B 367 (1991) 60.
  * [3] C. G. Callan, Jr., J. A. Harvey and A. Strominger, “Supersymmetric string solitons,” in: Proc. String Theory and Quantum Gravity ’91(Trieste, 1991) [arXiv:hep-th/9112030].
  * [4] M. F. Atiyah, N. J. Hitchin, V. G. Drinfeld and Y. I. Manin, Phys. Lett. A 65 (1978) 185.
  * [5] E. Witten, “Some Comments on the Recent Twistor Space Constructions,” Workshop on Applications of Complex Manifold Techniques to Problems in Theoretical Physics, 1978
  * [6] E. Corrigan and P. Goddard, “Construction of Instanton and Monopole Solutions and Reciprocity,” Annals Phys. 154 (1984), 253
  * [7] N. H. Christ, E. J. Weinberg and N. K. Stanton, “General self-dual Yang-Mills Solutions,” Phys. Rev. D 18 (1978), 2013
  * [8] S. Donaldson, “The ADHM construction of Yang-Mills instantons”, arXiv:2205.08639v1 [math.DG]
  * [9] E. Witten, “Sigma models and the ADHM construction of instantons,” J. Geom. Phys. 15 (1995) 215 [hep-th/9410052].
  * [10] N. D. Lambert, “Quantizing the (0,4) Supersymmetric ADHM Sigma Model,” Nucl. Phys. B 460 (1996), 221-232 [arXiv:hep-th/9508039].
  * [11] M. R. Douglas, “Gauge fields and D-branes”, J. Geom. Phys. 28 (1998), 255-262 [arXiv:hep-th/9604198].
  * [12] N. D. Lambert, “Heterotic p-branes from massive sigma models,” Nucl. Phys. B 477, 141-154 (1996) [arXiv:hep-th/9605010].
  * [13] N. D. Lambert, “D-brane bound states and the generalized ADHM construction,” Nucl. Phys. B 519, 214-224 (1998) [arXiv:hep-th/9707156].
  * [14] C. V. Johnson, “On the (0,4) conformal field theory of the throat,” Mod. Phys. Lett. A 13, 2463-2474 (1998) [arXiv:hep-th/9804201].
  * [15] A. Ali and M. Ilahi, “Complementary ADHM Instanton Sigma Model,” [arXiv:2305.05951 [hep-th]].
  * [16] A. Ali and M. Ilahi, “$Z_{2}$ Symmetry of $AdS_{3}\times S^{3}\times S^{3}\times S^{1}$ Superstrings,” [arXiv:2306.13970 [hep-th]].
  * [17] A. Ali, “Superalgebra Doubling in $AdS_{3}$ Superstrings,” [arXiv:2306.11047 [hep-th]].
  * [18] E. Witten, “Some Comments on String Dynamics” [hep-th/9507121], in Strings ’95, Future Perspectives in String Theory, ed. I. Bars et. al.
  * [19] A. Ali, M. Ilahi and S. R. Thottoli, “Quantization of Complementary ADHM Sigma Model,” [arXiv:2306.10002 [hep-th]].
  * [20] A. Giveon, D. Kutasov and N. Seiberg, “Comments on String Theory on $AdS_{3}$,” Adv. Theor. Math. Phys. 2 (1998) 733 [hep-th/9806194].
  * [21] M. Ademollo, L. Brink, A. D’Adda, R. D’Auria, E. Napolitano, S. Sciuto, E. D. Giudice, P. D. Vecchia, S. Ferrara, F. Gliozzi, R. Musto and R. Pettorino, “Supersymmetric Strings and Color Confinement,” Phys. Lett. B 62 (1976) 105.
  * [22] M. Ademollo, L. Brink, A. D’Adda, R. D’Auria, E. Napolitano, S. Sciuto, E. D. Giudice, P. D. Vecchia, S. Ferrara, F. Gliozzi, R. Musto, R. Pettorino and J. Schwarz, “Dual String with $U(1)$ Color Symmetry,” Nucl. Phys. B 111 (1976) 77.
  * [23] A. Sevrin, W. Troost and A. van Proeyen, “Superconformal Algebras in Two-Dimensions with N=4”, Phys. Lett. B 208 (1988) 447.
  * [24] E. A. Ivanov, S. O. Krivonos and V. M. Leviant, “Quantum N=3, N=4 Superconformal WZW Sigma Models,” Phys. Lett. B 215 (1988) 689 Erratum: [Phys. Lett. B 221 (1989) 432]
  * [25] A. Ali, “Conformal Symmetry of Superstrings on $AdS_{3}\times S^{3}\times T^{4}$ and D1 / D5 system,” Mod. Phys. Lett. A 17 (2002) 2477 [hep-th/0007021].
  * [26] A. Ali, “Free Field Realizations of N=4 Superconformal Algebras,” Indian J. Pure Appl. Phys. 38 (2000), 446-452
  * [27] A. Ali, “Types of Two-dimensional N = 4 Superconformal Field Theories,” Pramana 61 (2003), 1065-1078 [arXiv:hep-th/9906096].
  * [28] A. Ali and A. Kumar, “A New N=4 Superconformal Algebra,” Mod. Phys. Lett. A 8, 1527-1532 (1993) [arXiv:hep-th/9301010].
  * [29] Z. Hasiewicz, K. Thielemans and W. Troost, “Superconformal Algebras and Clifford Algebras,” J. Math. Phys. 31, 744 (1990)


would not be an upper bound of $B$. -

Assume $(q_{n})_{n\in\mathbb{N}}$ is monotonically increasing. Take
$\lambda^{\prime}$ to be the least upper bound of
$Ran((q_{n}\lambda)_{n\in\mathbb{N}})$. Take any $\Delta\lambda>0$. By (A)
there exists an $n\in\mathbb{N}$ s.t. for all $i,j>n$,
$q_{j}\lambda\in(q_{i}\lambda-\Delta\lambda,q_{i}\lambda+\Delta\lambda)$. If
follows that for all $i>n$,
$q_{i}\lambda\in(\lambda^{\prime}-\Delta\lambda,\lambda^{\prime}+\Delta\lambda)$.

By (B), the proof for monotonically decreasing sequences is similar. ∎

It is a foundational result of real analysis that all Cauchy-convergent
sequences of rational numbers converge to a real number, and for all real
numbers there exist monotonic, Cauchy-convergent sequences of rational numbers
that converge to it.

######  Theorem 200.

If $(q_{n})_{n\in\mathbb{N}}$ and $(q_{n}^{\prime})_{n\in\mathbb{N}}$ are two
monotonic sequences of rational numbers that converge to the same real number
then $\lim(q_{n}\lambda)=\lim(q_{n}^{\prime}\mathbf{\lambda})$.

###### Proof.

Assume $(q_{n})_{n\in\mathbb{N}}$ and $(q_{n}^{\prime})_{n\in\mathbb{N}}$ are
monotonically increasing. Because they converge to the same real number, they
have the same least upper bound. $(q_{n}\lambda)_{n\in\mathbb{N}}$ and
$(q_{n}^{\prime}\lambda)_{n\in\mathbb{N}}$ must then also have the same least
upper bound. By the proof of Thm 199, they have the same limit.

All other cases are similar.∎

######  Definition 201.

If $(q_{n})_{n\in\mathbb{N}}$ is a monotonic, Cauchy-convergent sequence of
rational numbers and $\lim q_{n}=r$ then for any $\lambda\in\Lambda$,
$r\lambda\equiv\lim(q_{n}\lambda)$.

By Thm 200 the above definition uniquely defines multiplication by a real
number.

######  Theorem 202.

If $\Lambda$ is a dense parameter and $\mathbf{1}$ any element of $\Lambda$
that greater than $0$

1) For any real number $r$, $r\mathbf{1}\in\Lambda$

2) For any $\lambda\in\Lambda$, there exists a real number, $r$, s.t.
$r\mathbf{1}=\lambda$

###### Proof.

1) Follow immediately from Thm 199.

2) We’ll take the case of $\lambda>0$; $\lambda<0$ is similar and $\lambda=0$
is trivial.

A: If $\lambda>0$ then there exists a rational number $q$ s.t
$0<q\mathbf{1}<\lambda$

\- The greatest lower bound of the set of $\frac{1}{2^{n}}\mathbf{1}$,
$n\in\mathbb{N}^{+}$, is $0$; since $\lambda>0$ there must be an $i$ s.t.
$\frac{1}{2^{i}}\mathbf{1}<\lambda$ -

Take $q$ to be any rational number s.t. $q\mathbf{1}<\lambda$. By Thm 191
there’s an $n\in\mathbb{N}^{+}$ s.t. $nq\mathbf{1}>\lambda$; take $m_{0}$ to
be the smallest such element of $\mathbb{N}^{+}$. Take
$\lambda_{0}=(m_{0}-1)q\mathbf{1}$; note that $\lambda_{0}\leq\lambda$ and
$\lambda-\lambda_{0}\leq q\mathbf{1}$. Similarly for each $i\in\mathbb{N}$
take $m_{i}$ to be the smallest element of $\mathbb{N}^{+}$ s.t.
$m_{i}(\frac{q}{2^{i}})\mathbf{1}>\lambda$. With
$\lambda_{i}\equiv(m_{i}-1)(\frac{q}{2^{i}})\mathbf{1}$ it follows that
$\lambda_{i}\leq\lambda_{i+1}\leq\lambda$ and
$\lambda_{i}-\lambda\leq(\frac{q}{2^{i}})\mathbf{1}$.

Consider the sequence
$((m_{i}-1)(\frac{q}{2^{i}})\mathbf{1})_{n\in\mathbb{N}}$; for any
$\Delta\lambda$ there’s an $n$ s.t. $(\frac{q}{2^{n}})<\Delta\lambda$. For all
$j>n$,
$\lambda-(m_{j}-1)(\frac{q}{2^{j}})\mathbf{1}<(\frac{q}{2^{n}})\mathbf{1}<\Delta\lambda$,
so $((m_{i}-1)(\frac{q}{2^{i}})\mathbf{1})_{n\in\mathbb{N}}$ converges to
$\lambda$. with $r=\lim(m_{i}-1)(\frac{q}{2^{i}})$, $\lambda=r\mathbf{1}$.∎

######  Theorem 203.

If $\Lambda$ is a dense parameter and $\mathbf{1}$ any element of $\Lambda$
that greater than $0$

1) For $r_{1},r_{2}\in\mathbb{R}$,
$r_{1}\lambda+r_{2}\lambda=(r_{1}+r_{2})\lambda$

2) For $r_{1},r_{2}\in\mathbb{R}$, $\lambda>0$, if $r_{1}>r_{2}$ then
$r_{1}\lambda>r_{2}\lambda$

###### Proof.

1) If $(q_{n})_{n\in\mathbb{N}}$ and $(q_{n}^{\prime})_{n\in\mathbb{N}}$ are
monotonically increasing sequences of rational numbers and $\lim q_{n}=r_{1}$
and $\lim q_{n}^{\prime}=r_{2}$ then $\lim(q_{n}+q_{n}^{\prime})=r_{1}+r_{2}$.

2) Follows from (1) and opa 7, and the fact that $r_{3}\equiv r_{2}-r_{1}$ is
a positive real number. ∎

Thms 202 & 203 create a complete characterization of dense parameters.


# Help the Blind See: Assistance for the Visually Impaired through Augmented
Acoustic Simulation

Alexander Mehta
Theia Project
Sunnyvale, CA
<EMAIL_ADDRESS>
Ritik Jalisatgi
Theia Project
Sunnyvale, CA
<EMAIL_ADDRESS>

###### Abstract

An estimated 253 million people have visual impairments. These visual
impairments affect everyday lives, and limit their understanding of the
outside world. This can pose a risk to health from falling or collisions. We
propose a solution to this through quick and detailed communication of
environmental spatial geometry through sound, providing the blind and visually
impaired the ability to understand their spatial environment through sound
technology. The model consists of fast object detection and 3D environmental
mapping, which is communicated through a series of quick sound notes. These
sound notes are at different frequencies, pitches, and arrangements in order
to precisely communicate the depth and location of points within the
environment. Sounds are communicated in the form of musical notes in order to
be easily recognizable and distinguishable. A unique algorithm is used to
segment objects, providing minimal accuracy loss and improvement from the
normal $O(n^{2})$ to $O(n)$ (which is significant, as N in point clouds can
often be in the range of $10^{5}$). In testing, we achieved an R-value of
0.866 on detailed objects and an accuracy of 87.5% on an outdoor scene at
night with large amounts of noise. We also provide a supplementary video demo
of our system.

##  1 Introduction

An estimated 253 million have some visual impairment, with 36 million
completely blind people (Ackland et al. (2017)). Large risks are posed to
those with visual impairment – most notability falling and inability to avoid
obstacles due to little spatial understanding (Steinman et al. (2011); de Boer
et al. (2004); Gillespie et al. (2003)). According to Crews et al. (2016),
46.7% of blind persons over the age of 65 reported a fall, which can result in
death or severe injury. Solutions to these problems have taken shape in
various forms (reported on in Prior Work section). Our solution involves using
the visually impaired’s audio understanding and Pavlovian Conditioning to
address this crisis.

##  2 Background

####  2.0.1 Motivation

The idea of echolocation has been commonly known as an attribute of animals
such as bats, but the ability to echolocate extends to humans. Particularly,
blind or visually impaired humans can possess traits of echolocation according
to a small (N=37) preliminary study by Thaler (2013). This is confirmed by
Thaler and Goodale (2016) which finds that echolocation can be a strong
alternative to sight for the blind. This is backed up by Thaler et al. (2011)
who measured brain data to confirm that blind individuals that echolocate
effectively often use the same places of their brain stimulated by visual
activity. The paper also confirms an important assumption – echolocation from
both external (such as environmental sounds) and internal assistance (such as
headphones) has similar accuracy for blind users. Due to this, many devices
have been made to assist with echolocation, but none precisely communicate
spatial information such as the size of objects near the user.

####  2.0.2 Prior Work

Several systems have been created in order to address blindness through audio
techniques.

Object Detection and Segmentation based assistance by Jiang et al. (2016)
addresses this through a real time system to provide users with descriptions
of objects that are played at where the objects are located. This is similar
to the technique presented in this paper, except our technique relies on
properties of sound instead of text to audio descriptions and gives much more
detailed information about the user’s environmental spatial geometry.
Additionally, Jiang et al. (2016) connects to a large server for computation,
while the new system runs offline on a Raspberry Pi 4 and an Intel RealSense
Camera, allowing for less constraints on operation in-the-wild.

Text-based approaches have been introduced by Sarwar et al. (2022) by using
OCR technology in order to describe signs to the blind. This technology can
run on a raspberry pi, but fails to account for the overall scene, and non
text markers.

This paper presents an innovative approach to enhance spatial communication
through sound by utilizing a two-step strategy. The first is encoding spatial
geometry data into a series of sounds (Pitch Based Depth Perception), and the
second is performing object-segmentation on the spatial geometry data through
floodfill to track potentially important areas. It combines the efficiency
Sarwar et al. (2022) while describing large amounts of information similar to
Jiang et al. (2016)

##  3 Materials and Methods

The method proposed in this paper addresses the challenge of spatial
communication through sound by utilizing a two-part approach. The first is
mapping spatial geometry data into a series of sounds (Pitch Based Depth
Perception), and the second is performing object-segmentation on the spatial
geometry data through floodfill to track potentially important areas.

###  3.1 Pitch Based Depth Perception

We address communicating the locations of points in the user’s environment
through multiple properties of sound: pitch, pan, and index. We communicate
the dimension of depth through pitch, with lower pitched notes indicating
further depths, then we address the horizontal dimension through panning (the
distribution of a sound’s volume between the left and right headphone), and
finally we address the vertical dimension through the index in which a note
was communicated. When used all together, the locations of multiple points
within the user’s environment may be interpreted which allows the user to
mentally map out their spatial environment.

Desmos Link: here

Depth to note function:
$96-2\cdot\operatorname{floor}\left(range\cdot\left(\frac{\left(x-start\right)}{\left(start-
end\right)}\right)^{0.8}\right)\left\\{end\geq x\geq start\right\\}$ We use
this function to convert the depth in meters to the pitch of a MIDI note
(higher values mean higher pitch).

The function was designed to use the full distinguishable pitch spectrum, and
for changes in depth to be more distinguishable at closer values than further
values.

Original Image Depth Camera Perception Chunking/ Downsampling Depth to MIDI
function applied Play Sound in specific order

We take the RGBD 2D image array provided by the camera, and then downsample it
through nearest neighbor sampling to 16 x 12. Then, using this downsampled
RGBD array, we create a MIDI note array by using our depth-note function to
convert each depth value in the downsampled array, to a MIDI note value. Each
note is also panned between the left and right sides of the user’s headphones,
with the leftmost value being panned all the way to the left, and the
rightmost value all the way to the right. Finally, to communicate this MIDI
note array, we play each MIDI note in each column of the array in order from
top to bottom, and iterate columns from right to left.

In this study, we employed object detection techniques, as outlined in the
subsequent section, to enhance the user’s object differentiation ability. The
implementation entailed assigning the objects in a scene with unique volume
levels. This approach enabled the user to gain a more comprehensive
understanding of the environment, through the provision of auditory cues
regarding the presence of various objects. It should be noted that this
feature was not included in our field demo, as we found that rapid movement
could result in disorienting changes due to the volume shifts. These findings
will be further explored in the limitations and future work section.

###  3.2 Proposed Floodfill Algorithm

To segment objects within our scene, we use an O(n) floodfill based algorithm
that segments the objects based on their relative positions to other points in
the pointcloud. We use this floodfill algorithm as a basis to provide the user
with better understanding of the objects themselves.

A naive floodfill based segmentation is performed by taking points,
calculating distance to other points, and filling the nearest points based on
a threshold. Clusters of close points will then constitute an object. Below is
a pseudo code version of the naive solution.

Data: Pointcloud representation of space

Result: Segmented Object Pointclouds

declare distance threshold;

while  _Points in Pointcloud not assigned to object_ do

declare targetPoint;

for  _point in Pointcloud_ do

if  _$\sqrt{point^{2}+targetPoint^{2}} <threshold$_ then

place in current object bucket;

end if

end for

choose new point to perform pointcloud algorithm on;

end while

Algorithm 1 Naive Algorithm

This naive solution performs slowly. Taking $O(n)$ to iterate through N points
in a pointcloud. Then for each point, the operation of calculating distances
to other points takes $O(n)$, resulting in an $O(n^{2})$ algorithm. For RGB-D
image point clouds, N can be over $3*10^{5}$ (640 x 480). When N is squared it
can result in time complexity of over $9*10^{10}$ operations. This is
extremely slow, and in echolocation, which is expected to be instant, can be
detrimental to the user. Additionally, the operations are slow, as the
$\sqrt{(}x)$ function is slow to compute due to the fact it utilizes Newton’s
method1 which converges quadratically.

11footnotetext: https://encyclopediaofmath.org/index.php?title=Newton_method

In order to solve this, we introduce an algorithm that works in $O(n)$.

We employ a chunking based cache to remove distance calculations. For each
point, we round the $x$,$y$, and $z$ to the nearest multiple of $1/n$ for each
dimension. These rounded values are considered to be a chunk. This way, a
cluster of points in the pointcloud will already be in the same chunk / group.
We then create a lookup table where keys are the original coordinates, and the
value is the chunk. We can then do the $O(n)$ floodfill that exists when
filling a 2D grid, except in 3D. We simply flood the chunks in the same
fashion one would flood squares in a grid. This leaves us with groups of
chunks, which we then use to determine groups of points by converting the
chunks in each group to their stored points.

Data: Pointcloud representation of space

Result: Segmented Object Pointclouds

declare chunk cache; for  _point in Pointcloud_ do

assign point to specific chunk;

end for

declare set of unflooded cache buckets; while  _set contains points_ do

declare select point and assign to the first item in set;

if  _surrounding cache buckets contain point_ then

mark those points as part of the current object bucket;

remove from unflooded set;

end if

end while

Algorithm 2 Optimized Object Detection Algorithm

This algorithm is able to perform floodfill without calculating relative
distances and reducing the complexity of the algorithm to $O(n)$. For the
average pointcloud size of $3*10^{5}$. Our algorithm runs in $3*10^{5}$
operations instead of $9*10^{10}$. Additionally, each operation is
significantly faster with the removal of the $sqrt(x)$ function. Below is a
visual representation.

Raw Pointcloud Data 3D space is separated into chunks Chunks are floodfilled
Points assigned to object that was assigned to respective chunk

###  3.3 Implementation

We implement our model in both C# with Unity and Python with PyGame. We use
the Unity implementation for standardized tests and the lightweight python
version for in the wild inference.

The PyGame version for in-the-wild demos and inference doesn’t use object
detection, as users in a moving scene may feel disoriented by the fast paced
change of objects if too many objects enter and exit a scene (discussed in the
limitations section). Additionally, the Unity implementation takes use of
Unity’s 3D audio protocol in order to effectively convey to the user where a
sound comes from. A limitation of this algorithm is headphones often don’t
provide high quality 3D audio, specifically the ability to distinguish the
height of sounds is lacking.

###  3.4 Physical System

To receive spatial input, we use an Intel Realsense D415 depth camera which
provides depth information with low compute resources. To run our model
portably, we use a Raspberry Pi 4+ ARM based SoC running Ubuntu 22.01. We fit
our model onto the user like a headlamp, with the camera being strapped to the
users forehead, and computations being ran on a Raspberry Pi 4+ housed in a 3D
case which is strapped to the back of the user’s head. Power banks of all
sizes may be used to power the device, and can be swapped out for a smaller or
larger one if necessary.

###  3.5 Experiments

####  3.5.1 Object Segmentation Evaluation

In order to evaluate the object segmentation accuracy of the proposed
algorithm in real-world scenarios, we evaluated the performance of segmenting
basic elementary shapes from one another. The accuracy of the proposed $O(n)$
algorithm was tested and compared to a $O(n^{2})$ baseline algorithm. The
running time was measured in milliseconds (ms) and total basic compute
operations, and the correct number of segmented objects was evaluated by
comparing the difference between the actual and expected objects in the scene
using Pearson’s Correlation Coefficient, which allows us to measure
effectiveness even if the amount of objects detected is not accurate. For
example, if the algorithm detected 5 objects instead of 6, the 5 detected
would still count towards the score. Additionally, we use a percent accuracy
metric to measure the number of times the algorithm ran perfectly. This real
world, in-the-wild evaluation method provides a comprehensive assessment of
the algorithm’s performance in real-world scenarios compared to the base
algorithm.

Pearson’s Correlation Coefficient (referenced to as the R-value)

$PCC=\frac{{}\sum_{i=1}^{n}(x_{i}-\overline{x})(y_{i}-\overline{y})}{\sqrt{\sum_{i=1}^{n}(x_{i}-\overline{x})^{2}(y_{i}-\overline{y})^{2}}}$

####  3.5.2 Pitch Based Depth Perception

Due to the interpretive nature of the visual to audio based algorithm, we
provide in-the-wild demonstrations to provide a comprehensive evaluation of
the proposed approach in real-world scenarios (link here).

##  4 Results

####  4.0.1 Object Detection Results

We evaluated our model on the Intel RealSense in-the-wild outdoor pointcloud
set. We find that the dataset accurately depicted the scene, while having
minor noise issues as times when light conditions were unideal.

Black and White Image captured by depth camera (left) and our Unity object
representation (right). The algorithm detected the correct amount of items in
the correct places. Specifically the 3 human cutouts are seen from left to
right and the wall behind.

As seen, our algorithm falls short when it comes to small artifacts, believing
small artifacts are actually objects.

We evaluated the performance of our algorithm using pointcloud files provided
by Intel, containing up to 30 seconds of pointcloud data per file.
Additionally, we also tested the algorithm on object files from the dataset
presented in the paper by Zhou and Koltun (2013). Results were collected from
three distinct datasets: an outdoor scene captured by a D415 RealSense camera,
known for its ability to generate large and accurate pointclouds; an indoor
scene captured by a SR300, a lower quality depth camera; and the object
dataset by Zhou and Koltun (2013), which includes high-quality pointclouds of
various scenes, not recorded using RealSense technology.

Object Detection Results Data Description Pearsons R Accuracy (%) Type Outdoor
Scene N/a 87.5% High Quality RealSense with IR Ball 0.316 50% Low Quality
RealSense Zhou and Koltun (2013) 0.866 12.5% High Object Count Note that
Pearsons R is considered N/a if the object count doesn’t change

####  4.0.2 Pitch Based Depth Perception

The present study offers a demonstration of Pitched Based Depth Perception,
which can be accessed through the following link:
https://www.youtube.com/watch?v=YrfPQbwcvGg. It is widely acknowledged that
pitch perception is a variable trait among individuals and can be improved
through training (Van Hedger et al. (2019)). As a result, it is not possible
to accurately evaluate the effectiveness of this approach to depth perception
without conducting a comprehensive evaluation involving a large sample of
human participants. This will be tested in future study, as noted in the
future work section.

Visualization of pitch based algorithm from video demo, where the lighter the
color, the greater the pitch.

More noisy, complex, example

##  5 Discussion

The present study demonstrates that the accuracy of our algorithm in dark
outdoor scenes is high and remains robust in the presence of large artifacts,
with an accuracy of 87.5%. This is of particular significance for individuals
with visual impairments, as navigating dark environments can be particularly
challenging. Furthermore, research has shown that partial blindness is more
prevalent during night time hours Bijveld et al. (2013). The high level of
accuracy achieved in this study may be attributed to the use of high-quality
point clouds, suggesting that the algorithm benefits from a greater amount of
data. These findings provide motivation for future research to invest in
higher quality materials in real-world environments in order to achieve
optimal results.

The results of the present study indicate that the accuracy of the algorithm
on the ball dataset is relatively low, as evidenced by the R-value of 0.316
and an accuracy rate of 50%. These findings are consistent with previous
claims that the performance of the algorithm is directly correlated with the
quality of the data used. Specifically, the use of a lower quality realsense
camera appears to have led to a reduction in accuracy. This serves as further
evidence of the importance of utilizing high-quality data in order to achieve
optimal performance with this algorithm.

The results of the present study indicate that the dataset provided by Zhou
and Koltun (2013) yielded high R-value results but relatively low accuracy
results. This is a reasonable outcome, given that the dataset used in this
study included a larger number of objects in a scene compared to previous
datasets, with a size range of $10^{1}$ to $10^{2}$. While it may be
unrealistic to expect perfect recall in such a scenario, the high R-value
values suggest that the algorithm demonstrates a high level of precision, even
if the overall accuracy is not perfect.

##  6 Conclusion

In this study, we proposed a novel approach for providing audio-based
assistance to individuals with visual impairments. Our approach comprises of
two main algorithms, namely a depth perception algorithm and a chunk-based
floodfill algorithm. The depth perception algorithm allows the visually
impaired to perceive object depth and location in three dimensions through the
use of different pitches of sounds played through headphones. The second
algorithm is a chunk based floodfilling algorithm. We find that this algorithm
improves from the base time complexity of $O(N^{2})$ to $O(N)$ through our
caching mechanism. We find this algorithm performed well in scenes, but falls
short in low quality pointclouds. Overall, we find a large research
opportunity in audio based scene understanding to help those who are visually
impaired.

####  6.0.1 Limits, Implications, and Future Work

Our findings indicate that the performance of the algorithm is limited by the
quality of the depth camera on which it is run. As such, future research
should focus on developing the algorithm’s robustness to lower quality
environments. Additionally, it should be noted that the proposed algorithm is
unique in nature and therefore, it is not straightforward to establish a
benchmark for comparison with other algorithms in the field. Specifically, the
algorithm is exclusively focused on object detection for non-visual, audio-
based understanding, making it difficult to determine the required level of
accuracy for effective translation from visual to audio-based understanding.

Our evaluation revealed that the depth based perception algorithm functioned
as intended, however, it has the major implication of requiring the user to
possess a sufficient level of auditory acuity to comprehend the range of
pitches employed by the algorithm. In order to fully evaluate the
effectiveness of this approach, future research should include human
participant testing. Specifically, this testing should focus on assessing the
users’ ability to comprehend visual scenes through their hearing, in order to
determine the minimum level of auditory acuity required for the effective use
of this algorithm.

We find that different audio based cue algorithms for depth perception are
limited by headphone designs. Due to the lackluster quality of 3D spatial
audio headphones, the ability to best convey the scene is hindered by current
audio technology. Future improvements to headphone technology will lead to a
similar improvement in the abilities of our algorithm to distinguish different
directions. Specifically, the ability to differentiate sounds vertically would
allow the algorithm to be extended to understanding where sounds were being
played from in all 3 dimensions without overwhelming the user with too many
sound changes.

In this study, we proposed a solution for providing audio-based assistance to
individuals with visual impairments, who constitute an estimated 253 million
people worldwide. These visual impairments can significantly affect everyday
lives, limiting their understanding of the outside world and posing a risk to
their health from falling or collisions. Our solution aims to enhance the
mobility and independence of visually impaired individuals by providing quick
and detailed communication of environmental spatial geometry through sound.
The proposed model consists of fast object detection and 3D environmental
mapping, which is communicated through a series of quick sound notes that
convey the depth and location of points within the environment. The sounds are
communicated in the form of musical notes to make them easily recognizable and
distinguishable. A unique algorithm was used to segment objects, resulting in
minimal accuracy loss and significant improvement in computational efficiency
from the standard $O(n^{2})$ to $O(n)$. In testing, we achieved an R-value of
0.866 on detailed objects and an accuracy of 87.5% in an outdoor scene at
night with large amounts of noise. The results of this study demonstrate the
potential of audio-based assistance in augmenting the mobility and
independence of visually impaired individuals.

## References

  * Ackland et al. (2017) Peter Ackland, Serge Resnikoff, and Rupert Bourne.  World blindness and visual impairment: despite many successes, the problem is growing.  _Community Eye Health_ , 30(100):71–73, 2017\.
  * Steinman et al. (2011) Bernard A Steinman, Anna Q D Nguyen, Jon Pynoos, and Natalie E Leland.  Falls-prevention interventions for persons who are blind or visually impaired.  _Insight_ , 4(2):83–91, 2011.
  * de Boer et al. (2004) Michiel R de Boer, Saskia MF Pluijm, Paul Lips, Annette C Moll, Hennie J Völker-Dieben, Dorly JH Deeg, and Ger HMB van Rens.  Different aspects of visual impairment as risk factors for falls and fractures in older men and women.  _Journal of bone and Mineral Research_ , 19(9):1539–1547, 2004.
  * Gillespie et al. (2003) Lesley D Gillespie, William J Gillespie, M Clare Robertson, Sarah E Lamb, Robert G Cumming, and Brian H Rowe.  Interventions for preventing falls in elderly people.  _Cochrane database of systematic reviews_ , (4), 2003.
  * Crews et al. (2016) John E. Crews, Chiu-Fung Chou, Judy A. Stevens, and Jinan B. Saaddine and.  Falls among persons aged $\geq$65 years with and without severe vision impairment — united states, 2014.  _MMWR. Morbidity and Mortality Weekly Report_ , 65(17):433–437, May 2016.  doi: 10.15585/mmwr.mm6517a2.  URL https://doi.org/10.15585/mmwr.mm6517a2.
  * Thaler (2013) Lore Thaler.  Echolocation may have real-life advantages for blind people: an analysis of survey data.  _Frontiers in Physiology_ , 4, 2013.  ISSN 1664-042X.  doi: 10.3389/fphys.2013.00098.  URL https://www.frontiersin.org/articles/10.3389/fphys.2013.00098.
  * Thaler and Goodale (2016) Lore Thaler and Melvyn A Goodale.  Echolocation in humans: an overview.  _Wiley Interdisciplinary Reviews: Cognitive Science_ , 7(6):382–393, 2016.
  * Thaler et al. (2011) Lore Thaler, Stephen R. Arnott, and Melvyn A. Goodale.  Neural correlates of natural human echolocation in early and late blind echolocation experts.  _PLOS ONE_ , 6(5):1–16, 05 2011.  doi: 10.1371/journal.pone.0020162.  URL https://doi.org/10.1371/journal.pone.0020162.
  * Jiang et al. (2016) Rui Jiang, Qian Lin, and Shuhui Qu.  Let blind people see: Real-time visual recognition with results converted to 3d audio.  03 2016.
  * Sarwar et al. (2022) Savera Sarwar, Muhammad Turab, Danish Channa, Aisha Chandio, M. Uzair Sohu, and Vikram Kumar.  Advanced audio aid for blind people, 2022.  URL https://arxiv.org/abs/2212.00004.
  * Zhou and Koltun (2013) Qian-Yi Zhou and Vladlen Koltun.  Dense scene reconstruction with points of interest.  _ACM Transactions on Graphics_ , 32, 07 2013.  doi: 10.1145/2461912.2461919.
  * Van Hedger et al. (2019) S. C. Van Hedger, S. L. M. Heald, and H. C. Nusbaum.  Absolute pitch can be learned by some adults.  _PLoS One_ , 14(9):e0223047, 2019.
  * Bijveld et al. (2013) Mieke M. C. Bijveld, Maria M. van Genderen, Frank P. Hoeben, Amir A. Katzin, Ruth M. A. van Nispen, Frans C. C. Riemslag, and Astrid M. L. Kappers.  Assessment of night vision problems in patients with congenital stationary night blindness.  _PLOS ONE_ , 8(5):1–11, 05 2013.  doi: 10.1371/journal.pone.0062927.  URL https://doi.org/10.1371/journal.pone.0062927.


# Developing High Quality Training Samples for Deep Learning Based Local
Climate Zone Classification in Korea

Minho Kim*
Seoul National University
<EMAIL_ADDRESS>
Doyoung Jeong*
Seoul National University
<EMAIL_ADDRESS>
Hyoungwoo Choi*
Seoul National University
<EMAIL_ADDRESS>
Yongil Kim
Seoul National University
<EMAIL_ADDRESS>
Department of Civil and Environmental Engineering

###### Abstract

Two out of three people will be living in urban areas by 2050, as projected by
the United Nations, emphasizing the need for sustainable urban development and
monitoring. Common urban footprint data provide high-resolution city extents
but lack essential information on the distribution, pattern, and
characteristics. The Local Climate Zone (LCZ) offers an efficient and
standardized framework that can delineate the internal structure and
characteristics of urban areas. Global-scale LCZ mapping has been explored,
but are limited by low accuracy, variable labeling quality, or domain
adaptation challenges. Instead, this study developed a custom LCZ data to map
key Korean cities using a multi-scale convolutional neural network. Results
demonstrated that using a novel, custom LCZ data with deep learning can
generate more accurate LCZ map results compared to conventional community-
based LCZ mapping with machine learning as well as transfer learning of the
global So2Sat dataset.

##  1 Introduction

The Local Climate Zone (LCZ) classification framework was introduced as a
climate-based classification of urban-rural areas for temperature studies [1].
The LCZ scheme (see Appendix, Figure 5) is composed of 17 urban and natural
classes based on regions of uniform surface cover, structure, material, and
anthropogenic activity [2]. In general, LCZs are culturally-neutral and are
characterized by screen-height temperature apparent over dry surfaces on calm,
clear nights in areas of simple relief [1]. Based on this concept, the
integration of remote sensing data and data-driven methods have enabled global
LCZ classification at 100 m spatial resolution [3].

###  1.1 Global Trends in LCZ Classification

The World Urban Database and Access Portal Tools (WUDAPT) is a community-based
project which provided an automated LCZ classification protocol to produce LCZ
maps of global cities using the random forest (RF) classifier. [3, 4]. Recent
studies have extended LCZ mapping to national and continental scales, such as
the United States [2], Europe [5], Hong Kong [6], and major cities in China
[7]. Another approach in literature was improving the classification accuracy
when employing deep learning over conventional machine learning algorithms [8,
9, 10, 11]. Furthermore, Zhu et al. (2019) introduced the large-scale So2Sat
dataset which is comprised of Sentinel-1 and 2 image patches with
corresponding LCZ labels for 42 global cities. The dataset has been tested and
benchmarked using sophisticated deep learning models, demonstrating superior
classification accuracy results over conventional methods [12, 13]. However,
the WUDPAT protocol is limited to low classification accuracy and poor
generalization ability [8, 12, 14], while the So2Sat dataset requires advanced
domain adaptation techniques to fully harness its potential.

###  1.2 Deep Learning Based LCZ Mapping for Korea

This study aims to develop LCZ mapping for the Korean peninsula by developing
LCZ training samples of major Korean cities and classifying using a multi-
scale convolutional neural network (MSCNN). The major contributions of this
study can be summarized by the following points:

(1) High-quality LCZ training samples for Korea: Adopted sampling method [3,
12] for scene classification via deep learning [8, 10, 11] to produce
fundamental training samples for Korea.

(2) Assessment of classification accuracy: Evaluated classification accuracy
using RF and MSCNN models with the custom LCZ training samples and transfer
learning of the state-of-the-art So2Sat dataset [12], thereby investigating
the effectiveness of custom and global LCZ datasets

##  2 Data and Methods

###  2.1 Input Data

The sampling phase requires a basemap, natural cover, and built-up cover
layers. The specifications of each data type is displayed in Table 1.
Sentinel-2 images were used as the main basemap and were acquired using the
Copernicus Open Hub. Google Earth Engine cloud-free mosaics can also be used
[15]. The Normalized Difference Vegetation Index (NDVI) is applied as the
natural cover layer and the master building information (MBI) dataset,
provided by the Electronic Architectural Administration Information System
(EAIS), is used as the built-up cover. The MBI dataset includes building
information of each major city in Korea such as building height, number of
floors, area, and location. These input layers can be combined to help label
sampling (see Appendix, Figure 6).

Table 1: List of input data specifications Data | Main Function | Source
---|---|---
Sentinel-2 | Basemap and input (Bands 1 - 10) | Copernicus Open Hub
NDVI | Natural cover (LCZ A - LCZ G) | Sentinel-2 Bands 3 and 4
MBI Building height | Built-up cover (LCZ 1 - LCZ 6) | EAIS

###  2.2 Sampling Workflow

This study adopted a scene classification approach to create 32 by 32 pixel
patches based on Liu et al. (2020) and to match the So2Sat’s patch size. The
entire labeling workflow (see Appendix, Figure 7) is based on the workflow in
Zhu et al. (2019). In this study, the LCZ labels are sampled as points which
act as the patch’s centroid [10, 11, 12]. Larger image patches tend to improve
classification accuracy due to local information [8, 10, 11]. Second, the LCZ
labels are guided using MBI and reviewed with Google Earth and Google Street
View to ensure correctness and completeness [12]. Finally, once a sufficient
volume of training samples is accumulated, data augmentation is processed to
modify class weights to alleviate the effect of imbalanced classes.

###  2.3 Assessment of Classification Accuracy

First, the RF is used to represent the WUDAPT method. Second, MSCNN is built
with a multi-scale layer which uses three convolution layers with different
kernel sizes (3, 5, 7 pixels), followed by five convolution layer blocks in
order of convolution (CONV), batch normalization, ReLU activation function,
and max pooling layers. A dropout out of 25% was added after the first fully-
connected (FC) layer. MSCNN used the adam optimizer [16], batch size of 96,
learning rate of 0.002, decay factor of 0.004, and early stop of 15. Third,
the So2Sat dataset uses the benchmark ResNet for training [12]. The original
layers are frozen, while two fully-connected dense layers are added to the end
of the model for transfer learning [17].

Figure 1: Model architecture of MSCNN showing the multi-scale layer and
convolutional layers.

##  3 Results and Discussion

###  3.1 Contribution 1: High-quality LCZ training samples for Korea

The distribution of training samples for major Korean cities are provided in
Figure 2 with respect to each LCZ class. The different class proportions offer
a glimpse of each city’s urban form and structure. This list is not exhaustive
and we plan to continue increasing data volume and labeling quality. These
training samples can then be used as the fundamental training sample dataset
to generate a nation-level LCZ map of Korea.

Figure 2: Current distribution of labelled LCZ training samples of key cities.

###  3.2 Contribution 2: Classification Accuracy

LCZ classification was conducted for Seoul to produce 100 m resolution LCZ
maps using the three models mentioned in Section 2.3. As organized in Table 2,
MSCNN using the custom LCZ data outperformed all other methods with a superior
overall accuracy of 83.88%. Compared to RF when using the custom data
(74.27%), MSCNN demonstrates that deep learning is significant to improve
accuracy, while the low accuracy of So2Sat (54.42%) implies that using MSCNN
with the custom data is more effective than transfer learning a global
dataset, in spite of the large-scale data volume.

Table 2: LCZ classification accuracy Model | Overall | Kappa
---|---|---
So2Sat | 54.42% | 48.60%
RF | 74.27% | 41.00%
MSCNN | 83.88% | 64.99%
Figure 3: Class-wise comparison of F1-score results.

The F1-score results shown in Figure 3 indicate that MSCNN classified the
highly heterogeneous features more effectively compared to RF and the So2Sat
method. For instance, the built-up LCZ classes, such as LCZ 1 or LCZ 4 were
more defined using MSCNN, especially in densely compact urban areas and even
suburban and rural communities. The So2Sat method was able to classify built-
up areas like LCZ 1, but transfer learning of the global dataset introduced
confusion of natural cover in densely vegetated areas. This result can be
attributed to the difference in crop fields and covers between global cities
and Korea, which is particularly prevalent by seasonal variations. All of the
models struggled to classify similar LCZ classes (below 50% F1-score), such as
LCZ 2 and 3 as well as for LCZ 5 and 6 [9]. Future studies can integrate DSMs
or building height information from GIS layers such as OpenStreetMap and the
EAIS dataset.

Figure 4: Comparison of LCZ maps of Seoul for May 3, 2017 for So2Sat, RF, and
MSCNN.

##  4 Outlook and Future Work

Deep learning based LCZ classification methods are required to effectively
monitor urban areas and to produce high quality LCZ maps that delineate the
heterogeneous urban features effectively. Based on the progress of generating
global-scale LCZ maps and datasets in recent studies, this study developed
fundamental, custom data for LCZ mapping in Korea. The study’s results using
MSCNN confirm that deep learning outperforms the shallow RF classifier used by
WUDAPT, and that creating a novel, yet optimized labelled training sample data
can be more efficient compared to transfer learning the large-scale So2Sat
dataset. Future works will focus on increasing data volume and enhancing label
quality, while exploring ways to integrate global datasets such as So2Sat more
effectively. City-by-city comparisons of LCZ classification will also be
conducted to investigate the generalization ability of the models and training
sample data.

## References

  * [1] Ian D Stewart and Tim R Oke.  Local climate zones for urban temperature studies.  Bulletin of the American Meteorological Society, 93(12):1879–1900, 2012.
  * [2] Matthias Demuzere, Steve Hankey, Gerald Mills, Wenwen Zhang, Tianjun Lu, and Benjamin Bechtel.  Combining expert and crowd-sourced training data to map urban form and functions for the continental us.  Scientific data, 7(1):1–13, 2020.
  * [3] Benjamin Bechtel, Paul J Alexander, Jürgen Böhner, Jason Ching, Olaf Conrad, Johannes Feddema, Gerald Mills, Linda See, and Iain Stewart.  Mapping local climate zones for a worldwide database of the form and function of cities.  ISPRS International Journal of Geo-Information, 4(1):199–219, 2015\.
  * [4] Benjamin Bechtel, Paul J Alexander, Christoph Beck, Jürgen Böhner, Oscar Brousse, Jason Ching, Matthias Demuzere, Cidália Fonte, Tamás Gál, Julia Hidalgo, et al.  Generating wudapt level 0 data–current status of production and evaluation.  Urban climate, 27:24–45, 2019.
  * [5] Matthias Demuzere, Benjamin Bechtel, Ariane Middel, and Gerald Mills.  Mapping europe into local climate zones.  PloS one, 14(4):e0214474, 2019.
  * [6] Ran Wang, Chao Ren, Yong Xu, Kevin Ka-Lun Lau, and Yuan Shi.  Mapping the local climate zones of urban areas by gis-based and wudapt methods: A case study of hong kong.  Urban climate, 24:567–576, 2018.
  * [7] Chao Ren, Meng Cai, Xinwei Li, Lei Zhang, Ran Wang, Yong Xu, and Edward Ng.  Assessment of local climate zone classification maps of cities in china and feasible refinements.  Scientific reports, 9(1):1–11, 2019.
  * [8] Cheolhee Yoo, Daehyeon Han, Jungho Im, and Benjamin Bechtel.  Comparison between convolutional neural networks and random forest for local climate zone classification in mega urban areas using landsat images.  ISPRS Journal of Photogrammetry and Remote Sensing, 157:155–170, 2019.
  * [9] Chunping Qiu, Michael Schmitt, Lichao Mou, Pedram Ghamisi, and Xiao Xiang Zhu.  Feature importance analysis for local climate zone classification using a residual convolutional neural network with multi-source datasets.  Remote Sensing, 10(10):1572, 2018.
  * [10] Shengjie Liu and Qian Shi.  Local climate zone mapping as remote sensing scene classification using deep learning: A case study of metropolitan china.  ISPRS Journal of Photogrammetry and Remote Sensing, 164:229–242, 2020.
  * [11] Johannes Rosentreter, Ron Hagensieker, and Björn Waske.  Towards large-scale mapping of local climate zones using multitemporal sentinel 2 data and convolutional neural networks.  Remote Sensing of Environment, 237:111472, 2020.
  * [12] Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthias Häberle, Yuansheng Hua, Rong Huang, et al.  So2sat lcz42: A benchmark dataset for global local climate zones classification.  arXiv preprint arXiv:1912.12171, 2019.
  * [13] Chunping Qiu, Xiaochong Tong, Michael Schmitt, Benjamin Bechtel, and Xiao Xiang Zhu.  Multi-level feature fusion-based cnn for local climate zone classification from sentinel-2 images: Benchmark results on the so2sat lcz42 dataset.  arXiv preprint arXiv:2005.07983, 2020.
  * [14] Benjamin Bechtel, Matthias Demuzere, Panagiotis Sismanidis, Daniel Fenner, Oscar Brousse, Christoph Beck, Frieke Van Coillie, Olaf Conrad, Iphigenia Keramitsoglou, Ariane Middel, et al.  Quality of crowdsourced data on urban morphology—the human influence experiment (huminex).  Urban Science, 1(2):15, 2017.
  * [15] Michael Schmitt, Lloyd H Hughes, Chunping Qiu, and Xiao Xiang Zhu.  Aggregating cloud-free sentinel-2 images with google earth engine.  ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences, 4, 2019.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [17] Bei Zhao, Bo Huang, and Yanfei Zhong.  Transfer learning with fully pretrained deep convolution networks for land-use classification.  IEEE Geoscience and Remote Sensing Letters, 14(9):1436–1440, 2017\.

##  5 Appendix

The following figures are included to help users understand and replicate our
sampling and classification procedure. Despite the difficulty of manually
labeling training samples, we believe that optimizing and enhancing the input
data is essential to ensure high quality LCZ maps.

Figure 5: Overview of the LCZ framework with a description of each class. For
more detailed explanations, readers are directed to more comprehensive studies
[1, 3]. Figure 6: Example of a stacked layer analysis for LCZ sampling. Figure
7: Detailed decision rule workflow for built-up LCZ class labeling adapted
from Zhu et al. (2019). The main difference is the addition of the EAIS
database to help differentiate building height as well as the consideration of
pervious and impervious covers when classifying low-rise and industry-based
LCZ classes.