# Building Extraction from Remote Sensing Images via an Uncertainty-Aware
Network

Wei He, , Jiepan Li, Weinan Cao, Liangpei Zhang, , Hongyan Zhang W. He, J. Li,
W. Cao, and L. Zhang are with the State Key Laboratory of Information
Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan
430072, China ({weihe1990, jiepanli, jackzhang, zlp62}@whu.edu.cn).H. Zhang is
with the School of Computer Science, China University of Geosciences, Wuhan,
430074, PR China (zhanghongyan@whu.edu.cn).

###### Abstract

Building extraction aims to segment building pixels from remote sensing images
and plays an essential role in many applications, such as city planning and
urban dynamic monitoring. Over the past few years, deep learning methods with
encoder–decoder architectures have achieved remarkable performance due to
their powerful feature representation capability. Nevertheless, due to the
varying scales and styles of buildings, conventional deep learning models
always suffer from uncertain predictions and cannot accurately distinguish the
complete footprints of the building from the complex distribution of ground
objects, leading to a large degree of omission and commission. In this paper,
we realize the importance of uncertain prediction and propose a novel and
straightforward Uncertainty-Aware Network (UANet) to alleviate this problem.
Specifically, we first apply a general encoder–decoder network to obtain a
building extraction map with relatively high uncertainty. Second, in order to
aggregate the useful information in the highest-level features, we design a
Prior Information Guide Module to guide the highest-level features in learning
the prior information from the conventional extraction map. Third, based on
the uncertain extraction map, we introduce an Uncertainty Rank Algorithm to
measure the uncertainty level of each pixel belonging to the foreground and
the background. We further combine this algorithm with the proposed
Uncertainty-Aware Fusion Module to facilitate level-by-level feature
refinement and obtain the final refined extraction map with low uncertainty.
To verify the performance of our proposed UANet, we conduct extensive
experiments on three public building datasets, including the WHU building
dataset, the Massachusetts building dataset, and the Inria aerial image
dataset. Results demonstrate that the proposed UANet outperforms other state-
of-the-art algorithms by a large margin. The source code of the proposed UANet
is available at https://github.com/Henryjiepanli/Uncertainty-aware-Network.

###### Index Terms:

Building extraction, remote sensing, uncertainty-aware

##  I Introduction

Building extraction aims to distinguish building footprints from high-
resolution remote sensing (RS) images, and has made remarkable progress in the
past few decades. Owing to its potential applications, building extraction has
also been extended to various fields, such as city planning [1], urban dynamic
monitoring [2], and disaster detection [3].

(a) Image

(b) GT

(c) SOTA

(d) Ours

Figure 1: Uncertainty visualizations between our proposed UANet and the state-
of-the-art (SOTA) method for building extraction (BuildFormer [28]). (c) and
(d) are achieved by the operation $0.5-|0.5-\star|$, with $\star$ representing
the output of the $Sigmoid$ function.

Up to date, numerous studies have made significant contributions to the
extraction of buildings from high-resolution remote sensing (RS) images ([30,
28, 36]). Compared to middle/low resolution RS images, the higher-resolution
RS images provide more detailed information about ground objects, while also
increasing intra-class variances and decreasing inter-class variances, posing
various challenges to accurately extract building footprints [4]. To overcome
the aforementioned challenges, research on building extraction has undergone a
long-term development. In the early stage, a major effort was devoted to the
design of more distinctive features. For example, [5] utilized multiple colors
and color-invariant spaces to select the representative corners and chose some
corner candidates to generate the rooftop outline. Based on information about
entropy and color, [6] introduced texture information to differentiate between
buildings and trees. Moreover, [7] firstly took advantage of the contour
driven by edge-flow to extract the building boundary, and then segmented the
compositional polygons of the building roof by Joint Systems Engineering Group
(JSEG). Nevertheless, due to the limited robustness and representativeness,
the aforementioned hand-crafted features cannot handle the complex correlation
between the buildings and the background.

In the past few years, deep learning algorithms have been successfully applied
to RS building extraction and have become the mainstream technical tools.
Initially, in order to adopt deep learning algorithms into building extraction
research, some simple networks were proposed based on patch-based annotation.
[9] designed a neural network consisting of three convolutional layers and two
fully connected layers to achieve the automatic extraction of buildings. [10]
designed a patch-based convolutional neural network (CNN) architecture that
replaced fully connected layers with global average pooling (GAP) to improve
extraction performance. However, the patch-based classification method has two
unavoidable drawbacks [11], namely, a huge computational burden and limited
long-distance information exchange. As a result, these methods cannot fully
exploit contextual information in high-resolution RS images, making it
difficult to completely and accurately extract buildings from complex
backgrounds.

Figure 2: The structure of the Uncertainty-Aware Network, which is composed of
a general encoder-decoder, a prior information guide module (PIGM), and an
uncertainty-aware fusion module (UAFM).

Fully Convolutional Network (FCN) [12] is a landmark pixel-based segmentation
method that provides an encoder-decoder architecture, which has become a
paradigm. In detail, the encoders process the input image to generate multi-
level features, and the decoders adopt various strategies to output the
semantic results. Currently, typical backbone networks, such as VGG [13],
ResNet [14], ResNext [15], Res2Net [16], and even some networks based on
transformers [17, 18], are selected as encoders. After obtaining hierarchical
features from the encoder, a sequence of decoder structures is proposed. For
designing the decoders, the general strategy is to take advantage of multi-
level encoded features from the aspects of modeling multi-scale contextual
information [22, 31, 32, 33, 34, 63], mining long-range dependency information
[21, 23, 41, 30, 64, 65], or feature refinement [27, 25, 62].

Regarding the decoding strategies for modeling multi-scale contextual
information, two typical plug-and-play modules, namely, Atrous Spatial Pyramid
Pooling (ASPP) [22] and Receptive Field Block (RFB) [31], have been proposed.
Furthermore, [32] enhanced the extraction of local features with a reasonable
stacking of small-dilation-rate dilation convolutions, thereby effectively
reducing the cases of ambiguous results for small-sized building segmentation.
[33] proposed a novel Adaptive Screening Feature Network to teach the network
to adjust the receptive field and adaptively enhance useful feature
information. Moreover, [34] utilized a graph-based scale-aware structure to
model and reason the interactions between different scale features.

Regarding the decoding strategies for mining long-range dependency
information, there have been many great works on the design of both encoders
and decoders. Given that CNN is limited to local connections, some researchers
have replaced CNNs with transformers in the design of encoders. The current
transformer networks, such as Swin Transformer [17], Pyramid Vision
Transformer [18], and so on, have all proved their strength in capturing long-
distance information. Additionally, some work has introduced unique modules to
establish long-distance contextual information in the decoders. For example,
an Asymmetric Pyramid Non-Local Block [19] was introduced by [20] to extract
contextual global information. [21] combined an ASPP [22] and a Non-Local
Block [23] to propose a pyramidal self-attentive module for convenient
embedding in the network. [30] took advantage of a local-global dual-stream
network to adaptively capture local and long-range information for accurate
building extraction.

Regarding the decoding strategies for feature refinement, researchers are
suggested to model the long distances and accurately locate spatial locations,
which is often overlooked by CNN due to the spatial transformation invariance.
Therefore, some works have utilized boundaries and contours to refine the
final segmentation. [25] proposed the Feature-Pairwise Conditional Random
Field based on the Graph Convolutional Network (GCN) [26], which is a
conditional random field for pairs of potential pixels with local constraints,
incorporating the feature maps extracted by the CNN. [29] analyzed the
conflict between deep CNN downsampling operation and accurate boundary
segmentation, and introduced a Gated GCN into the CNN structure to generate
clear boundaries and high fine-grained pixel-level classification results.
Furthermore, [27] designed a boundary refinement module (BR) to progressively
refine the prediction of the building by perceiving and refining the edges of
the building. Although taking the boundary information into account seems to
be an appropriate way to refine the details of the segmentation, the richness
of the boundary samples can also be an important factor that should not be
ignored to limit the performance exploration of such methods.

In summary, great progress has been made for high resolution RS building
extraction using deep-learning-based methods. However, due to the complex
distribution of ground objects in RS images and diverse appearances of
buildings, current decoding strategies will inevitably produce
misunderstanding of the building, resulting in uncertain prediction, which is
clearly reflected in Fig. 1c. As analyzed in [48], the reason why current
decoding strategies fail in some difficult cases is that they lack enough
attention to hard-to-segment samples. Especially in RS images, some buildings
are not salient enough and do not appear frequently, which will result in
uncertainty of the model. Therefore, solving the uncertain prediction is the
key to further improving the performance of building extraction model. In
fact, uncertainty-aware learning has been studied in the general segmentation
[56, 57, 52] and detection [58] areas. At the beginning, the uncertainty
analysis is always tight to complex networks (Bayesian deep learning [48, 42]
$,etc.,$) with a huge computational cost. Subsequently, the general frameworks
(Probabilistic Representational Network [51, 52], Generative Adversarial
Network (GAN) $,etc.$) are designed to improve the prediction certainty.
However, when dealing with a building extraction task, these models/frameworks
may fail to explore the characteristics of RS and result in unsatisfied
results.

In this paper, we realize the importance of building uncertainty prediction,
and propose the Uncertainty-Aware Network (UANet). The proposed UANet can
automatically rank the background uncertainty and building uncertainty of RS,
and progressively guide the attention to these uncertain pixels during the
interaction of features. In detail, the proposed UANet first adopts a
conventional encoder-decoder structure to output multi-level features and a
relatively uncertain extraction map. On the basis of these results, we attempt
to further solve the uncertainty problem and divide the following process into
two key parts. At the beginning, we put forward a prior information guide
module (PIGM) via a novel cross-attention mechanism to realize the enhancement
of both spatial and channel aspects. Then, we propose the uncertainty-aware
fusion module (UAFM) and innovatively invent an uncertainty rank algorithm
(URA) to realize the elimination of uncertainty as much as possible. As shown
in Figs. 1c and 1d, compared with BuildFormer, our UANet shows less
uncertainty particularly around the edges. The main contributions of this
study are as follows.

  1. 1.

We introduce the uncertainty concept to building extraction and propose the
UANet that can maintain high certainty faced with diverse scales, complex
backgrounds, and various building appearances, $etc$.

  2. 2.

We put forward a novel feature refinement way named PIGM from both spatial and
channel aspects.

  3. 3.

We propose the UAFM and the URA to relieve the uncertainty condition and
achieve a refined extraction map with low uncertainty.

The rest of this paper is organized as follows. In Section 2, we analyze and
introduce the structure and components of our UANet. The experiments and
results analysis are presented in Section 3, the ablation study of our
proposed modules is given in Section 4, and the conclusions are outlined in
Section 5.

##  II Methodology

###  II-A Overview

Aiming to eliminate the uncertainty of the final extraction map as much as
possible, we propose the Uncertainty-Aware Network (UANet). As shown in Fig.2,
we first adopt a general encoder-decoder network to get a relatively uncertain
extraction map. Regarding the general encoder-decoder part, we adopt VGG-16
[13] as the encoder backbone to extract multi-level features from the input
image, introduce a multi-branch dilation convolution blocks to enhance the
encoded features ($E_{i},\left\\{i=2,3,4,5\right\\}$), and use a typical
cross-fusion strategy (Feature Pyramid Network (FPN [50])) to obtain a
relatively uncertain extraction map $M_{5}$. Based on the output features
($F_{i},\left\\{i=1,2,3,4,5\right\\}$) and uncertain extraction map $M_{5}$,
our UANet acts as a decoder strategy to deal with the general building
extraction challenges and output a refined extraction map with low
uncertainty. In detail, we first put forward a prior information guide module
(PIGM) to take advantage of the prior information of the obtained extraction
map to enhance the highest-level feature. Subsequently, the uncertainty-aware
fusion module (UAFM) is utilized progressively to eliminate the uncertainty of
features from high level to low level. Finally, UANet outputs the final
refined extraction map with lower uncertainty.

###  II-B Prior Information Guide Module

In fact, the process to achieve the relatively uncertain extraction map
$M_{5}$ is a general decoding strategy, which cannot solve the current
uncertainty problem. However, we believe that the information provided by
$M_{5}$ is still very valuable. Therefore, to achieve a more accurate and less
uncertain prediction, we try to consider the extraction map $M_{5}$ as prior
knowledge and take advantage of it to realize the enhancement of the features.
As mentioned before, the highest-level feature with the largest dimension
lacks spatial information due to the smallest resolution. Based on this
consideration, we propose the Prior Information Guide Module (PIGM) to guide
the highest-level feature to realize refinement from both spatial and channel
aspects. As shown in Fig. 3, we first utilize $M_{5}$ to guide the highest-
level feature to learn the corresponding spatial relationships. Subsequently,
we continue to use $M_{5}$ to model the channel dependence of the enhanced
feature.

In detail, the inputs of PIGM are $F_{5}\in{\mathbb{R}}^{C\times H\times W}$
and $M_{5}\in{\mathbb{R}}^{1\times H\times W}$. At the beginning, we split the
input feature $F_{5}$ from the channel dimension and get $C$ feature maps
$F_{5}^{i}\in{\mathbb{R}}^{1\times H\times W}$:

$F_{5}^{i}=Split(F_{5}),i=1,2,...,C.$ (1)

On the one hand, we reshape $F_{5}^{i}$ to compress its dimension and obtain
$V_{5}^{i}\in{\mathbb{R}}^{C\times N}$ ($N=H\times W$). On the other hand, we
reshape and transpose $F_{5}^{i}$ to get $Q_{5}^{i}\in{\mathbb{R}}^{N\times
C}$:

$\begin{split}&V_{5}^{i}=Reshape(F_{5}^{i}),i=1,2,...,n,\\\
&Q_{5}^{i}=Transpose(Reshape(F_{5}^{i})),i=1,2,...,n.\end{split}$ (2)

Figure 3: The structure of the Prior Information Guide Module (PIGM).

Subsequently, we need to guide the input feature $F_{5}$ to learn the spatial
information by exploring the prior map $M_{5}$, so we reshape $M_{5}$ to
$K_{5}\in{\mathbb{R}}^{N\times C}$:

$\centering K_{5}=Reshape(M_{5}).\@add@centering$ (3)

Then, we perform the cross-attention, which conducts the matrix multiplication
between $Q_{5}^{i}$ and $K_{5}$ via Softmax function to obtain
$T_{5}^{i}\in{\mathbb{R}}^{N\times N}$ that represents the relationship
between each channel of $F_{5}$ and the prior map $P_{5}$:

$\centering\begin{split}&T_{5}^{i}=Softmax(Q_{5}^{i}\otimes
K_{5}),i=1,2,...,n.\end{split}\@add@centering$ (4)

We take use of the relation map $T_{5}^{i}$ to multiply $V_{5}^{i}$ and
achieve the enhanced feature map $O_{5}^{i}\in{\mathbb{R}}^{1\times H\times
W}$. Meanwhile, all the enhanced feature maps from $O_{5}^{1}$ to $O_{5}^{C}$
are concatenated to formulate $C_{5}$. At last, we can obtain the spatial
enhanced feature $R_{5}\in{\mathbb{R}}^{C\times H\times W}$ with a residual
structure:

$\centering\begin{split}&O_{5}^{i}=V_{5}^{i}\otimes T_{5}^{i},i=1,2,...,C,\\\
&C_{5}=Concat(O_{5}^{0},O_{5}^{i},...,O_{5}^{C}),\\\ &R_{5}=\alpha\times
C_{5}+F_{5},\end{split}\@add@centering$ (5)

where $\alpha$ is a learnable parameter.

From another perspective, we still attempt to utilize $M_{5}$ to achieve the
enhancement of $R_{5}$. As shown in Fig. 3, we reshape and transpose $M_{5}$
to obtain $Q_{5}^{{}^{\prime}}\in{\mathbb{R}}^{N\times 1}$:

$\centering\begin{split}&Q_{5}^{{}^{\prime}}=Transpose(Reshape(M_{5})).\end{split}\@add@centering$
(6)

At the same time, we reshape the spatial enhanced feature $R_{5}$ as
$K_{5}^{{}^{\prime}}\in{\mathbb{R}}^{C\times N}$, and use the matrix
multiplication among $Q_{5}^{{}^{\prime}}$ and $K_{5}^{{}^{\prime}}$ and the
Sigmoid function to get $S_{5}\in{\mathbb{R}}^{C\times 1}$. Afterward, by
multiplying $S_{5}$ and $R_{5}$ associated with a residual structure, we can
get the final feature $G_{5}\in{\mathbb{R}}^{C\times H\times W}$:

$\centering\begin{split}&K_{5}^{{}^{\prime}}=Reshape(R_{5}),\\\
&S_{5}=Sigmoid(Q_{5}^{{}^{\prime}}\otimes K_{5}^{{}^{\prime}}),\\\
&G_{5}=\beta\times S_{5}\times R_{5}+R_{5},\end{split}\@add@centering$ (7)

where $\beta$ is a learnable parameter.

###  II-C Uncertainty-Aware Fusion Module

In the previous stages, we successively acquire the uncertain extraction map
$M_{5}$ and the enhanced feature $G_{5}$. However, the uncertainty caused by
the intricate backgrounds and various scales still remains. Therefore, we
present the uncertainty-aware fusion module (UAFM) to tackle the high
uncertainty issue, as illustrated in Fig. 4.

As we all know, all the deep learning approaches output the extraction results
by using the $Softmax$ function to allocate the corresponding probability for
each pixel, which can be directly used to reflect the uncertainty of the model
in its predictions. As mentioned before, in RS images, some buildings are not
salient enough and do not appear frequently, which will result in the
uncertainty of model. To overcome such a uncertainty problem, we directly use
the $Sigmoid$ function to get the corresponding probabilities of all pixels in
the extraction map $M$ from spatial perspective, then we subtract all values
of the probability map with $0.5$ to measure the uncertainty belonging to
foreground ($U_{f}$) and meanwhile subtract $0.5$ with all values of the
probability map to measure the uncertainty belonging to background ($U_{b}$),

$\centering\begin{split}&U_{f}=Sigmoid(M)-0.5,\\\
&U_{b}=0.5-Sigmoid(M).\end{split}\@add@centering$ (8)

Subsequently, we rank the uncertainty of foreground and background into five
levels using the Uncertainty Rank Algorithm (URA), that is, the range of
$[-0.5,0)$ represents not in consideration (rank 0), the range of $[0,0.1)$
indicates the highest uncertainty (rank 5), the range of $[0.1,0.2)$
represents the relatively high uncertainty (rank 4), the range of $[0.2,0.3)$
represents the central uncertainty (rank 3), the range of $[0.3,0.4)$
indicates moderately low uncertainty (rank 2), and the range of $[0.4,0.5]$
denotes the lowest uncertainty (rank 1). We then assign corresponding
uncertainty levels as weights to the pixels, with the principle of attaching
higher weights to pixels with higher uncertainty, so as to pay more attention
on uncertain areas. We denote URA as:

$\mathbb{f}(i,j)=\left\\{\begin{array}[]{lr}\lfloor\frac{0.5-U_{{i,j}}}{0.1}\rfloor,U_{{i,j}}>=0,&\\\
0,U_{{i,j}}<0,&\end{array}\right.$ (9)

where $U_{{i,j}}$ means the pixel in $i_{th}$ row and $j_{th}$ column of
$U_{f}$ or $U_{b}$.

Figure 4: The structure of the Uncertainty-Aware Fusion Module (UAFM).

Therefore, after using URA to allocate the uncertainty level about the
uncertainty maps of the foreground and the background, respectively, we can
obtain the foreground uncertainty rank map ($R_{f}$) and the background
uncertainty rank map ($R_{b}$).

$\centering\begin{split}&R_{f}=URA(Sigmoid(M)-0.5),\\\
&R_{b}=URA(0.5-Sigmoid(M)),\end{split}\@add@centering$ (10)

We take the fusion of the highest $G_{5}$ and $F_{4}$ for example to
illustrate the whole fusion process. Specifically, the inputs of UAFM in this
layer are the enhanced feature $G_{5}$, the $F_{4}$, and the uncertain
extraction map $M_{5}$. Regarding the uncertainty-aware enhancement, on the
one hand, we apply URA to $M_{5}$ so that we can get the corresponding
foreground uncertainty rank map ($R_{f}^{5}$) and background uncertainty rank
map ($R_{b}^{5}$). Then we directly use $G_{5}$ to multiply with them to
highlight the uncertain pixels from both the foreground and background
perspectives. Subsequently, we concatenate these two enhanced features and
recover its original channel to get $G_{5}^{u}$ by a $1\times 1$ convolution
operation.

$\centering\begin{split}&G_{5}^{u}=Conv_{1\times
1}(Concat(R_{f}^{5}*G_{5},R_{b}^{5}*G_{5})),\end{split}\@add@centering$ (11)

On the other hand, we use the nearest neighbor interpolation method to
upsample $R_{f}^{5}$ and $R_{b}^{5}$ to the same size as $F_{4}$, and use the
same operation to highlight $F_{4}$ as the enhancement of $G_{5}$, and we can
get $F_{4}^{u}$.

$\centering\begin{split}&F_{4}^{u}=Conv_{1\times
1}(Concat(Up(R_{f}^{5})*F_{4},Up(R_{b}^{5})*F_{4})),\\\
\end{split}\@add@centering$ (12)

Finally, we upsample $G_{5}^{u}$ to match the size of $F_{4}^{u}$, concatenate
them together, and use a $3\times 3$ convolution operation to get the fused
feature $G_{4}$, which can output the less uncertain extraction map $M_{4}$ by
a $3\times 3$ convolution operation.

$\centering\begin{split}&G_{4}=Conv_{3\times
3}(Concat(F_{4}^{u},G_{5}^{u})),\\\ &M_{4}=Conv_{3\times
3}(G_{4}),\end{split}\@add@centering$ (13)

As shown in Fig. 4, we employ the uncertainty-aware fusion module (UAFM) to
fuse the features $G_{i}$ and $F_{i-1}$ layer-by-layer and decode the fused
feature to output the corresponding certainty-improved map $M_{i-1}$.

With such a UAFM, we can utilize $M_{4}$ to fuse $G_{4}$ and $F_{3}$ and
achieve output $M_{3}$, utilize $M_{3}$ to fuse $G_{3}$ and $F_{2}$ and
achieve output $M_{2}$, and utilize $M_{2}$ to fuse $G_{2}$ and $F_{1}$ and
output $M_{1}$, where $M_{1}$ can be viewed as the final refined extraction
map with the lowest uncertainty.

On the whole, we use the simple binary cross-entropy ($BCE$) loss function to
supervise all the outputs, and the overall loss is :

$\centering Loss=\sum_{i=1}^{5}BCE(M_{i},GT),\@add@centering$ (14)

where GT represents the ground truth.

##  III Experiments

###  III-A Dataset

To verify the superiority of our proposed UANet, we select three public
building extraction datasets to conduct extensive experiments, including the
WHU building dataset, the Massachusetts building dataset, and the Inria
building dataset. The detailed information of the whole three datasets is
described as follows:

  1. 1.

WHU building dataset [38] is composed of two types of images, $i.e.$,
satellite images, and aerial images. In our experimental settings, we only
conducted experiments on the aerial image dataset, which has $8,189$ image
tiles ($4,736$ tiles for training, $1,036$ tiles for validation, and $2,416$
tiles for testing). The spatial resolution is just $0.3m$, and the whole
aerial image dataset consists of $22,000$ buildings and covers a huge area of
over $450km^{2}$.

  2. 2.

Massachusetts building dataset [40] owns 151 aerial images of the Boston area
with spatial resolution $1m$. Composed of two types of scenes, $i.e.$, urban,
and suburban, the Massachusetts building dataset covers almost $340km^{2}$
areas, and all the image sizes are of $1500\times 1500$ pixels. The official
dataset contains a training set (137 images), a validation set (4 images), and
a testing set (10 images). We adopt some data augmentation ways to expand the
original training set to 411 images. For the training phase, we randomly crop
the images and labels into 1024 × 1024 pixels as input. And for both the
validating and testing phase, the images and labels are padded to the size of
1536 × 1536 pixels to ensure it is divisible by 32. It is worth mentioning
that we ignore the padding parts when computing evaluation metrics.

  3. 3.

Inria building dataset [39] contains 360 images collected from 5 cities
(Austin, Chicago, Kitsap, Tyrol, and Vienna). Referring to the official
suggestion, we select 1 to 5 tiles from each city for validation and the rest
for training. We first pad the original 5000 × 5000 images to 5120 × 5120
pixels and then crop them into 512 × 512 pixels image tiles. Second, we remove
the images without buildings, the remaining 9737 and 1942 image tiles used for
training and validation, respectively.

###  III-B Evaluation Metrics

TABLE I: Performance comparison with baseline models on the test datasets.
$\uparrow$ indicates the higher score the better and vice versa. The best
score for each metric is marked in red. The second score for each metric is
underlined.

Baseline | Year | WHU (%) | Massachusetts (%) | Inira (%)
---|---|---|---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
UNet # | 2015 | 85.92 | 92.39 | 91.78 | 93.01 | 68.48 | 81.47 | 80.99 | 81.96 | 74.40 | 85.32 | 86.39 | 84.28
HRNet # | 2019 | 85.64 | 92.27 | 91.69 | 92.85 | 69.39 | 81.93 | 81.49 | 82.38 | 75.03 | 85.73 | 86.56 | 84.92
MA-FCN | 2019 | 90.70 | 95.15 | 95.20 | 95.10 | 73.80 | 84.93 | 87.07 | 82.89 | 79.67 | 88.68 | 89.82 | 87.58
DSNet # | 2020 | 89.54 | 94.48 | 94.05 | 94.91 | 75.04 | 85.74 | 87.56 | 83.99 | 81.02 | 89.52 | 90.32 | 88.73
CBRNet | 2021 | 91.40 | 95.51 | 95.31 | 95.70 | 74.55 | 85.42 | 86.50 | 84.36 | 81.10 | 89.56 | 89.93 | 89.20
MSNet | 2022 | 89.07 | 93.96 | 94.83 | 93.12 | 70.21 | 79.33 | 78.54 | 80.14 | - | - | - | -
BOMSNet | 2022 | 90.15 | 94.80 | 95.14 | 94.50 | 74.71 | 85.13 | 86.64 | 83.68 | 78.18 | 87.75 | 87.93 | 87.58
LCS | 2022 | 90.71 | 95.12 | 95.38 | 94.86 | - | - | - | - | 78.82 | 88.15 | 89.58 | 86.77
BuildFormer # | 2022 | 90.73 | 95.14 | 95.15 | 95.14 | 75.03 | 85.73 | 86.69 | 84.79 | 81.24 | 89.71 | 90.65 | 88.78
BCTNet | 2023 | 91.15 | 95.37 | 95.47 | 95.27 | 75.04 | 85.74 | 87.57 | 83.99 | - | - | - | -
FD-Net | 2023 | 91.14 | 95.36 | 95.27 | 95.46 | 74.54 | 85.42 | 87.95 | 83.02 | - | - | - | -
Ours-UANet |  | 92.15 | 95.91 | 95.96 | 95.86 | 76.41 | 86.63 | 87.94 | 85.35 | 83.08 | 90.76 | 92.04 | 89.52

# means that the results were obtained by ourselves. The codes of other
compared methods are not released, we directly copy the results from the
original papers.

(a) Image

(b) GT

(c) UNet

(d) HRNet

(e) DSNet

(f) BuildFormer

(g) Ours

Figure 5: Visual Comparison on WHU building dataset.

To conduct a broad and comprehensive evaluation of our proposed model, we
chose four metrics, $i.e.$, intersection over union ($IoU$), F1 score ($F1$),
Precision, and Recall. At first, we use $TP$, $FP$, and $FN$ to represent the
true positive, the false positive, and the false negative, respectively. Then,
we give the definition of the four evaluation metrics as follows:

$\centering\begin{split}IoU=\frac{TP}{TP+FP+FN}\end{split}\@add@centering$
(15)
$\centering\begin{split}Precision=\frac{TP}{TP+FP}\end{split}\@add@centering$
(16)
$\centering\begin{split}Recall=\frac{TP}{TP+FN}\end{split}\@add@centering$
(17) $\centering\begin{split}F1=\frac{2\times Precision\times
Recall}{Precision+Recall}\end{split}\@add@centering$ (18)

###  III-C Experimental Settings

To comprehensively evaluate our proposed model, all related experiments are
implemented in PyTorch 1.8.1 (CUDA 11.1) on an NVIDIA GeForce RTX 3090 GPU
with 24GB of memory. In the training phase, we selected the AdamW [61]
optimizer and employed the cosine strategy to adjust the learning rate.
Additionally, we utilized random horizontal and vertical flipping to augment
the training data. According to the experimental settings in BuildFormer [28]
and our hardware conditions, for the WHU building dataset, we set the initial
learning rate to $10^{-3}$ and the batch size to 12. For the Massachusetts
building dataset, we set the initial learning rate to $5e^{-4}$ and the batch
size to 2. And for the Inria building dataset, we set the initial learning
rate to $5e^{-4}$ and the batch size to 12.

###  III-D Compared Methods

For a fair comparison, we selected two typical CNNs, $i.e.$, UNet[46] based on
VGG-16 [13], and HRNet[47] for the comparison. Meanwhile, we selected nine
state-of-the-art deep learning methods designed for building extraction,
$i.e.$, MA-FCN [36], DSNet [30], CBRNet[27], MSNet[34], BOMSNet[37], LCS[35],
BuildFormer[28], BCTNet [59], and FD-Net [60].

###  III-E Evaluation on WHU building dataset

####  III-E1 Quantitative Comparison

Table I lists the overall quantitative evaluation results of the different
methods obtained on the WHU building dataset. Compared with other SOTA
methods, our UANet can achieve the best performance on all metrics. In detail,
our UANet outperforms SOTA method CBRNet ( [27]) by $0.75$ percentage on the
$IoU$ metric, $0.40$ percentage on the $F1$ metric, $0.65$ percentage on the
$Precision$ metric, and $0.16$ percentage on the $Recall$ metric. The
significant advantages of these metrics reflect the superiority of our method,
proving that our proposed architecture with uncertainty consideration can
greatly improve the effect of building extraction.

(a) Image

(b) GT

(c) UNet

(d) HRNet

(e) DSNet

(f) BuildFormer

(g) Ours

Figure 6: Visual Comparison on Massachusetts building dataset.

####  III-E2 Visual Comparison

In order to compare our UANet with other SOTA methods more intuitively, we
visualize the extraction results of all methods. As shown in Fig. 5, the
qualitative results for UANet and the other methods on the WHU buildings
dataset are presented. For the first image, UNet, Deeplabv3+, HRNet, and
BuildFormer all fail to extract the building in the red circle, while DSNet
performs slightly better. By contrast, our UANet can accurately extract the
buildings in the pink circle, which is closer to the ground truth. For the
second image, all the compared methods wrongly recognize the road in the red
circle as the part of buildings, but our UANet avoids this problem perfectly.
Finally, for the third image, all the compared methods ignore the small
building in the red circle, but our UANet demonstrates its superiority over
the compared methods and successfully extracts this small building. It is
evident that the ignored buildings in the three examples above are in a
complex background, which leads to the uncertainty of the model. Faced with
such a situation, our UANet is able to achieve satisfactory results with less
uncertainty.

###  III-F Evaluation on Massachusetts building dataset

####  III-F1 Quantitative Comparison

Table I lists the overall quantitative evaluation results of the different
methods obtained on the Massachusetts building dataset. Compared with other
SOTA methods, our UANet can achieve the best performance on all metrics.
Specifically, compared with the SOTA method DSNet, our UANet can outperform it
by $1.37$ percentage on the IoU metric, $0.89$ percentage on the F1 metric,
$0.37$ percentage on the Precision metric, and $0.56$ percentage on the Recall
metric. Since the same backbone is used as other compared methods (except
BuildFormer), the huge advantage of our UANet indicates that our decoding
strategy is very effective.

####  III-F2 Visual Comparison

As shown in Fig. 6, we present three visual examples of all the compared
methods and our UANet on the Massachusetts building dataset. Due to the low
image resolution of the dataset and the dense distribution of buildings in the
image, it is evident that all the methods have a lot of errors in their
extraction results. However, it is obvious that our extraction result extracts
more details such as texture and edge than the compared methods, which is most
noticeable in the red box area. The more complex the environment, the better
our UANet performs than other compared methods, as our UANet can highlight the
uncertain areas and eliminate them to a large extent.

###  III-G Evaluation on Inria building dataset

####  III-G1 Quantitative Comparison

As shown in Table I, we list the overall quantitative evaluation results of
the different methods tested on the Inria building dataset. Compared with the
SOTA method BuildFormer, it is clear that our UANet can outperform it by
$1.84$ percentage on $IoU$, $1.05$ percentage on $F1$, $1.39$ percentage on
$Precison$, and $0.74$ percentage on $Recall$. This significant improvement
demonstrates the effectiveness of our approach to introduce uncertainty to
optimize decoding strategies.

(a) Image

(b) GT

(c) UNet

(d) HRNet

(e) DSNet

(f) BuildFormer

(g) Ours

Figure 7: Visual Comparison on Inria building dataset.

####  III-G2 Visual Comparison

As presented in Fig. 7, we select three typical examples to compare our UANet
with the other SOTA methods. In the first image, we can see that the buildings
in the red circle is covered by shadows from the buildings next to it, and the
compared methods fail to successfully extract the whole bodies of the
buildings. At the same time, compared with our result, there are still more
drawbacks. In the second image, we can easily find that the buildings in the
red circle are somewhat different from the other buildings around it, and
HRNet, DSNet, and BuildFormer ignore the real part of buildings but mistake
unrelated parts for buildings. By contrast, the result of our proposed UANet
is very close to the ground truth. In the third image, it is easy to find that
the compared methods mistakenly detect the part of the red rectangle as a
building, but our UANet succeed. From these three examples, it is convinced
that our UANet can make the right judgment in the face of complex
environments.

##  IV Ablation Study

In order to explore the effectiveness of our proposed modules in UANet, we
conduct extensive experiments on the three building datasets. We selected the
general encoder-decoder network used in our UANet as the baseline, which
utilizes VGG-16 as the encoder and use a conventional decoding method to
output an uncertain extraction map. Based on it, we verify the effectiveness
of the Prior Information Guide Module (PIGM), and the Uncertainty-Aware Fusion
Module (UAFM) in turn. In the following parts, we will give a detailed
analysis.

TABLE II: Ablation results on the test dataset. Baseline | PIGM | UAFM | WHU (%) | Massachusetts (%) | Inira (%)
---|---|---|---|---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
✓ |  |  | 87.35 | 92.90 | 92.25 | 93.56 | 69.73 | 82.17 | 85.41 | 79.16 | 79.08 | 88.32 | 87.77 | 88.88
✓ |  | ✓ | 91.25 | 95.43 | 95.73 | 95.13 | 74.84 | 85.61 | 87.56 | 83.75 | 81.84 | 90.01 | 90.43 | 89.61
✓ | ✓ | ✓ | 92.15 | 95.91 | 95.96 | 95.86 | 76.41 | 86.63 | 87.94 | 85.35 | 83.08 | 90.76 | 92.04 | 89.52

###  IV-A The effectiveness of PIGM

TABLE III: The ablation results about PIGM on the test dataset. Baseline | SC | CC | UAFM | WHU (%) | Massachusetts (%) | Inira (%)
---|---|---|---|---|---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
✓ |  |  | ✓ | 91.25 | 95.43 | 95.73 | 95.13 | 74.84 | 85.61 | 87.56 | 83.75 | 81.84 | 90.01 | 90.43 | 89.61
✓ | ✓ |  | ✓ | 91.02 | 95.27 | 95.22 | 95.32 | 75.03 | 85.92 | 87.78 | 84.14 | 82.61 | 90.48 | 91.68 | 89.30
✓ |  | ✓ | ✓ | 91.15 | 95.28 | 95.23 | 95.34 | 75.07 | 85.98 | 87.84 | 84.20 | 82.78 | 90.58 | 91.66 | 89.52
✓ | ✓ | ✓ | ✓ | 92.15 | 95.91 | 95.96 | 95.86 | 76.41 | 86.63 | 87.94 | 85.35 | 83.08 | 90.76 | 92.04 | 89.52

Guided by the uncertain extraction map $M_{5}$, we try to enhance the highest-
level features via PIGM. Different from the previous attention mechanism, we
introduced a cross-attention method, which helps the high dimensions features
to learn the spatial and the semantic relationship channel by channel. As
shown in Table II, by introducing the PIGM, the extraction accuracy can be
significantly improved. We conducted several experiments to verify the
detailed effect of the two components of the PIGM. As shown in Table III, to
verify the effectiveness of PIGM, we conducted four sets of experiments: 1)
without learning any correlation 2) just establishing the spatial correlation
(SC), 3) just establishing the channel correlation (CC), 4) establishing the
spatial and channel correlation in series (SC + CC). It is clear that the two
enhancement ways played their own role in the PIGM module.

###  IV-B The effectiveness of UAFM

The proposed UAFM can reduce the uncertainty of $G_{i}$ with the help of the
foreground uncertainty rank map $R_{f}^{i}$ and the background uncertainty
rank map $R_{b}^{i}$, and output feature $G_{i-1}$ with lower uncertainty. As
shown in Table II, we can easily find that the UAFM can bring significant
improvement of building extraction performance.

We also conducted extensive experiments to explore the accuracy improvement
brought about by such an uncertainty-aware strategy in detail. As presented in
Table IV, We set up four feature interaction methods: $Case1$: just
concatenate the adjacent layers of features and introduce deep supervision in
all levels; $Case2$: just use the $Sigmoid$ function to process the extraction
map from former level and utilize it achieve the feature interaction; $Case3$:
just use the foreground uncertainty ($R_{f}^{i}$) to achieve the feature
interaction; $Case4$: follow our proposed uncertainty-aware strategy which
utilizes both the foreground uncertainty ($R_{f}^{i}$) and the background
uncertainty ($R_{b}^{i}$) to achieve the feature interaction. It is evident
that the extraction accuracy is significantly improved with the guidance of
the uncertainty maps of both the foreground and the background, which can
intuitively reflect the huge advantage of our proposed strategy.

TABLE IV: The ablation results about UAFM on the test dataset. Baseline | WHU (%) | Massachusetts (%) | Inira (%)
---|---|---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
$Case1$ | 89.08 | 94.23 | 93.92 | 94.53 | 72.13 | 83.81 | 86.56 | 81.24 | 79.73 | 88.73 | 89.17 | 88.28
$Case2$ | 91.39 | 95.43 | 95.47 | 95.40 | 75.21 | 85.96 | 87.81 | 84.18 | 80.98 | 89.03 | 90.62 | 87.50
$Case3$ | 91.61 | 95.57 | 95.45 | 95.70 | 75.87 | 86.28 | 87.93 | 84.69 | 82.34 | 90.31 | 91.49 | 89.16
$Case4$ | 92.15 | 95.91 | 95.96 | 95.86 | 76.41 | 86.63 | 87.94 | 85.35 | 83.08 | 90.76 | 92.04 | 89.52
TABLE V: The results about $M_{i}$ on the test dataset. Baseline | WHU (%) | Massachusetts (%) | Inira (%)
---|---|---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$ | $IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
$M_{5}$ | 87.35 | 92.90 | 92.25 | 93.56 | 69.73 | 82.17 | 85.41 | 79.16 | 79.08 | 88.32 | 87.77 | 88.88
$M_{4}$ | 89.93 | 94.70 | 95.16 | 94.24 | 74.52 | 85.40 | 86.46 | 84.37 | 82.09 | 90.16 | 91.45 | 88.91
$M_{3}$ | 91.25 | 95.43 | 95.94 | 94.92 | 75.90 | 86.30 | 87.17 | 85.45 | 82.83 | 90.61 | 91.95 | 89.31
$M_{2}$ | 91.68 | 95.66 | 96.16 | 95.17 | 76.10 | 86.43 | 87.41 | 85.47 | 83.05 | 90.74 | 92.02 | 89.50
$M_{1}$ | 92.15 | 95.91 | 95.96 | 95.86 | 76.41 | 86.63 | 87.94 | 85.35 | 83.08 | 90.76 | 92.04 | 89.52

At the same time, In order to verify that UAFM can output feature $G_{i-1}$
and related prediction $M_{i-1}$ with lower uncertainty, we visualize
$G_{i-1}$ and the uncertainty reflected in $M_{i-1}$ of all levels. As
exhibited in Fig .8, we can observe that in each level, the enhanced features
$G_{i-1}$ can achieve cleaner objects and related edges compared to that of
$G_{i}$, and the uncertainty is progressively reduced. Besides, Table. V can
also illustrate the gradual enhancement of our high-to-low uncertain-aware
strategy from quantitative evaluation.

(a) Image/GT

(b) 5st level

(c) 4st level

(d) 3st level

(e) 2st level

(f) 1st level

Figure 8: Visual examples of building extraction. The first row represents the
visualizations of $G_{i}$, and the second row represents the uncertainty
visualization of $M_{i}$.

###  IV-C The analysis of URA

(a) Image/GT

(b) 5st level

(c) 4st level

(d) 3st level

(e) 2st level

(f) 1st level

Figure 9: Visual examples of building extraction. The first row represents the
visualizations of $R_{f}^{i}$, and the second row represents the uncertainty
visualization of $R_{b}^{i}$.

As the key algorithm in our UAFM, URA aims to rank the uncertainty level of
all pixels in the extraction map. As mentioned in Section II-C, the principle
of URA is to define a non-increasing linear function $\mathbb{f}$ from $U$ to
$R$. To simplify our design of $\mathbb{f}$, we define the uncertainty of
$0-0.5$ into five levels. To verify the effectiveness of our designed URA, we
visualize both $R_{f}^{i}$ and $R_{b}^{i}$ ($\left\\{i=1,2,3,4,5\right\\}$).
As shown in Fig. 9, we find that the level of uncertainty is decreasing
overall. We can conclude that assigning different weights to each level of
uncertainty can address the uncertainty problem to some extent. We also find
that, the weight of pixels with high uncertainty needs to be significantly
higher than that of pixels with low uncertainty.

###  IV-D The analysis of different encoders

TABLE VI: Ablation results about different encoders on the test dataset. Method | Inira (%)
---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
ResNet-50 | 82.17 | 90.21 | 91.24 | 89.20
Res2Net-50 | 83.17 | 90.81 | 91.89 | 89.76
PVT-V2-B2 | 83.34 | 90.91 | 91.86 | 89.97
VGG-16 | 83.08 | 90.76 | 92.04 | 89.52

As mentioned before, our UANet can be also used for other kinds of encoder-
decoder building extraction models to improve the certainty prediction. And we
select ResNet-50 [14], Res2Net-50 [16], VGG-16 [13] and PVT-V2-B2 [18] as
encoder-decoder backbones, to testify the efficacy of our UANet. As
illustrated in TableVI, we can easily find that our UANet can achieve
excellent results on different encoders, especially in the case of transformer
based architecture PVT-V2-B2. However, since most previous models utilize the
VGG-16 as the backbone, we also choose the same setting for a fair comparison.

###  IV-E The comparison with other uncertainty strategies

In our proposed UANet, we rank the uncertainty-level from both the foreground
and the background perspectives to reduce the uncertainty of features level by
level. To verify the superiority over other uncertainty strategies, we
compared our method with the uncertainty strategies used in other vision
tasks. In detail, on the one hand, we adopted the settings in [54] and added a
confidence estimation network to our VGG-16 based general encoder-decoder
structure to formalise the uncertainty as probability distribution over model
output and the input image. On the other hand, we followed the setting in [55]
and introduced the Conditional Variational Autoencoder (CAVE) to measure the
uncertainty of input data, which was followed by being input to our VGG-16
based general encoder-decoder structure with the input image. As illustrated
in TableVII, we can clearly see the superiority of our uncertainty strategy.
We believe that other uncertainty strategies do not take into account the
unique characteristics of the distribution of ground objects in RS images
(dense, small targets) and appear to be inapplicable. Relatively speaking, we
believe that the uncertainty in RS images is usually caused by insufficient
understanding of hard-to-segment buildings with less frequency in the process
of feature interaction, and our uncertainty-aware strategy can solve such a
problem perfectly.

TABLE VII: Ablation results about different uncertainty strategies on the test dataset. Method | Inira (%)
---|---
$IoU\uparrow$ | $F1\uparrow$ | $Pre\uparrow$ | $Recall\uparrow$
GAN | 79.14 | 88.36 | 88.82 | 87.60
CAVE | 80.55 | 89.09 | 89.62 | 88.56
Ours | 83.08 | 90.76 | 92.04 | 89.52

###  IV-F Complexity of UANet

In order to validate the efficiency of the proposed UANet, we compared the
amount of the parameters and the IoU on the Inria building dataset with the
current SOTA methods. As shown in Fig.10, our UANet achieves the highest
accuracy with the total parameter of 15.6 M, which is the lowest.

##  V Conclusion

Figure 10: Complexity and accuracy comparison of our UANet and the comparison
methods.

In this paper, we argue that the complex distribution of the ground objects,
inconsistent building scales, and various building styles bring some
uncertainty to the predictions of the general deep learning models, causing
the omission and the commission to a large extent. Therefore, we introduce the
concept of uncertainty and propose a novel uncertainty-aware network (UANet).
Firstly, we utilize a general encoder-decoder network to yield a general
uncertain extraction map. Secondly, we propose the PIGM to enhance the
highest-level features. Subsequently, the UAFM is proposed with the
uncertainty rank algorithm (URA) to eliminate the uncertainty of features from
high level to low level. Finally, the proposed UANet outputs the final
extraction map with lower uncertainty. By conducting sufficient experiments,
we validate the effectiveness of our UANet. The final high accuracy on three
public datasets indicates that the introduction of the uncertainty concept in
buildings has been extremely successful. However, although using such a way of
ranking the level of uncertainty can help us get a better extraction result,
how to allocate the weight adaptively of different uncertainty levels for URA
is still an unsolved problem in the paper, which will be a focus of our future
work.

## References

  * [1] M. M. Rathore, A. Ahmad, A. Paul, and S. Rho, “Urban planning and building smart cities based on the internet of things using big data analytics,” _Computer networks_ , vol. 101, pp. 63–80, 2016.
  * [2] S. Xu, X. Pan, E. Li, B. Wu, S. Bu, W. Dong, S. Xiang, and X. Zhang, “Automatic building rooftop extraction from aerial images via hierarchical rgb-d priors,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 56, no. 12, pp. 7369–7387, 2018.
  * [3] Z. Zheng, Y. Zhong, J. Wang, A. Ma, and L. Zhang, “Building damage assessment for rapid disaster response with a deep object-based semantic change detection framework: From natural disasters to man-made disasters,” _Remote Sensing of Environment_ , vol. 265, p. 112636, 2021.
  * [4] A. Bokhovkin and E. Burnaev, “Boundary loss for remote sensing imagery semantic segmentation,” in _International Symposium on Neural Networks_. Springer, 2019, pp. 388–401.
  * [5] M. Cote and P. Saeedi, “Automatic rooftop extraction in nadir aerial imagery of suburban regions using corners and variational level set evolution,” _IEEE transactions on geoscience and remote sensing_ , vol. 51, no. 1, pp. 313–328, 2012.
  * [6] M. Awrangjeb, C. Zhang, and C. S. Fraser, “Improved building detection using texture information,” _Int. Arch. Photogramm., Remote Sens. Spatial Inf. Sci._ , vol. 38, pp. 143–148, 2011.
  * [7] Y. Song and J. Shan, “Building extraction from high resolution color imagery based on edge flow driven active contour and jseg,” _IAPRSIS_ , vol. 37, pp. 185–190, 2008.
  * [8] H. Mayer, “Automatic object extraction from aerial imagery—a survey focusing on buildings,” _Computer vision and image understanding_ , vol. 74, no. 2, pp. 138–149, 1999.
  * [9] S. Saito, T. Yamashita, and Y. Aoki, “Multiple object extraction from aerial imagery with convolutional neural networks,” _Electronic Imaging_ , vol. 2016, no. 10, pp. 1–9, 2016.
  * [10] R. Alshehhi, P. R. Marpu, W. L. Woon, and M. Dalla Mura, “Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 130, pp. 139–149, 2017.
  * [11] L. Luo, P. Li, and X. Yan, “Deep learning-based building extraction from remote sensing images: A comprehensive review,” _Energies_ , vol. 14, no. 23, p. 7982, 2021.
  * [12] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [13] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [15] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1492–1500.
  * [16] S.-H. Gao, M.-M. Cheng, K. Zhao, X.-Y. Zhang, M.-H. Yang, and P. Torr, “Res2net: A new multi-scale backbone architecture,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 2, pp. 652–662, 2019\.
  * [17] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [18] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 568–578.
  * [19] Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai, “Asymmetric non-local neural networks for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 593–602.
  * [20] S. Wang, X. Hou, and X. Zhao, “Automatic building extraction from high-resolution aerial imagery via fully convolutional encoder-decoder network with non-local block,” _IEEE Access_ , vol. 8, pp. 7313–7322, 2020\.
  * [21] D. Zhou, G. Wang, G. He, T. Long, R. Yin, Z. Zhang, S. Chen, and B. Luo, “Robust building extraction for high spatial resolution remote sensing images with self-attention network,” _Sensors_ , vol. 20, no. 24, p. 7241, 2020.
  * [22] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [23] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7794–7803.
  * [24] L. Luo, P. Li, and X. Yan, “Deep learning-based building extraction from remote sensing images: A comprehensive review,” _Energies_ , vol. 14, no. 23, p. 7982, 2021.
  * [25] Q. Li, Y. Shi, X. Huang, and X. X. Zhu, “Building footprint generation by integrating convolution neural network with feature pairwise conditional random field (fpcrf),” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 11, pp. 7502–7519, 2020.
  * [26] J. You, J. Leskovec, K. He, and S. Xie, “Graph structure of neural networks,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 881–10 891.
  * [27] H. Guo, B. Du, L. Zhang, and X. Su, “A coarse-to-fine boundary refinement network for building footprint extraction from remote sensing imagery,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 183, pp. 240–252, 2022.
  * [28] L. Wang, S. Fang, X. Meng, and R. Li, “Building extraction with vision transformer,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–11, 2022.
  * [29] Y. Shi, Q. Li, and X. X. Zhu, “Building segmentation through a gated graph convolutional neural network with deep structured feature embedding,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 159, pp. 184–197, 2020.
  * [30] H. Zhang, Y. Liao, H. Yang, G. Yang, and L. Zhang, “A local-global dual-stream network for building extraction from very-high-resolution remote sensing images,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2020\.
  * [31] S. Liu, D. Huang _et al._ , “Receptive field block net for accurate and fast object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 385–400.
  * [32] R. Hamaguchi, A. Fujita, K. Nemoto, T. Imaizumi, and S. Hikosaka, “Effective use of dilated convolutions for segmenting small object instances in remote sensing imagery,” in _2018 IEEE winter conference on applications of computer vision (WACV)_. IEEE, 2018, pp. 1442–1450.
  * [33] J. Chen, Y. Jiang, L. Luo, and W. Gong, “Asf-net: Adaptive screening feature network for building footprint extraction from remote-sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–13, 2022.
  * [34] Y. Liu, Z. Zhao, S. Zhang, and L. Huang, “Multiregion scale-aware network for building extraction from high-resolution remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–10, 2022.
  * [35] Z. Liu, J. Ou, and Q. Shi, “Lcs: a collaborative optimization framework of vector extraction and semantic segmentation for building extraction,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2022.
  * [36] S. Wei, S. Ji, and M. Lu, “Toward automatic building footprint delineation from aerial images using cnn and regularization,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 58, no. 3, pp. 2178–2189, 2019.
  * [37] Y. Zhou, Z. Chen, B. Wang, S. Li, H. Liu, D. Xu, and C. Ma, “Bomsc-net: Boundary optimization and multi-scale context awareness based building extraction from high-resolution remote sensing imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–17, 2022.
  * [38] S. Ji, S. Wei, and M. Lu, “Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 57, no. 1, pp. 574–586, 2018.
  * [39] E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark,” in _2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_. IEEE, 2017, pp. 3226–3229.
  * [40] V. Mnih, “Machine learning for aerial image labeling,” Ph.D. dissertation, University of Toronto, 2013.
  * [41] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3146–3154.
  * [42] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [43] R. D. Soberanis-Mukul, N. Navab, and S. Albarqouni, “An uncertainty-driven gcn refinement strategy for organ segmentation,” _arXiv preprint arXiv:2012.03352_ , 2020.
  * [44] Z. Zheng and Y. Yang, “Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation,” _International Journal of Computer Vision_ , vol. 129, no. 4, pp. 1106–1120, 2021.
  * [45] T.-H. Vu, H. Jain, M. Bucher, M. Cord, and P. Pérez, “Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2517–2526.
  * [46] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [47] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 10, pp. 3349–3364, 2020.
  * [48] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [49] A. Li, J. Zhang, Y. Lv, B. Liu, T. Zhang, and Y. Dai, “Uncertainty-aware joint salient object and camouflaged object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 071–10 081.
  * [50] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [51] Y. Fang, H. Zhang, J. Yan, W. Jiang, and Y. Liu, “Udnet: Uncertainty-aware deep network for salient object detection,” _Pattern Recognition_ , vol. 134, p. 109099, 2023.
  * [52] F. Yang, Q. Zhai, X. Li, R. Huang, A. Luo, H. Cheng, and D.-P. Fan, “Uncertainty-guided transformer reasoning for camouflaged object detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 4146–4155.
  * [53] T. DeVries and G. W. Taylor, “Leveraging uncertainty estimates for predicting segmentation quality,” _arXiv preprint arXiv:1807.00502_ , 2018.
  * [54] J. Liu, J. Zhang, and N. Barnes, “Modeling aleatoric uncertainty for camouflaged object detection,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 1445–1454.
  * [55] N. Kajiura, H. Liu, and S. Satoh, “Improving camouflaged object detection with the uncertainty of pseudo-edge labels,” in _ACM Multimedia Asia_ , 2021, pp. 1–7.
  * [56] C. F. Baumgartner, K. C. Tezcan, K. Chaitanya, A. M. Hötker, U. J. Muehlematter, K. Schawkat, A. S. Becker, O. Donati, and E. Konukoglu, “Phiseg: Capturing uncertainty in medical image segmentation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22_. Springer, 2019, pp. 119–127.
  * [57] A. Jungo and M. Reyes, “Assessing reliability and challenges of uncertainty estimations for medical image segmentation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22_. Springer, 2019, pp. 48–56.
  * [58] F. Kraus and K. Dietmayer, “Uncertainty estimation in one-stage object detection,” in _2019 ieee intelligent transportation systems conference (itsc)_. IEEE, 2019, pp. 53–60.
  * [59] L. Xu, Y. Li, J. Xu, Y. Zhang, and L. Guo, “Bctnet: Bi-branch cross-fusion transformer for building footprint extraction,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–14, 2023.
  * [60] H. Guo, X. Su, C. Wu, B. Du, and L. Zhang, “Decoupling semantic and edge representations for building footprint extraction from remote sensing images,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023.
  * [61] I. Loshchilov and F. Hutter, “Fixing weight decay regularization in adam,” 2018\.
  * [62] J. Li, W. He, and H. Zhang, “Towards complex backgrounds: A unified difference-aware decoder for binary segmentation,” _arXiv preprint arXiv:2210.15156_ , 2022.
  * [63] L. Fang, P. Zhou, X. Liu, P. Ghamisi, and S. Chen, “Context enhancing representation for semantic segmentation in remote sensing images,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [64] X. Ding, C. Shen, T. Zeng, and Y. Peng, “Sab net: A semantic attention boosting framework for semantic segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [65] Q. Song, J. Li, H. Guo, and R. Huang, “Denoised non-local neural network for semantic segmentation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.


# Enhanced thermoelectric performance by van Hove singularities in the density
of states of type-II nodal-line semimetals

Nguyen T. Hung1,2<EMAIL_ADDRESS>Jyesta M. Adhidewata3,4 Ahmad
R. T. Nugraha4 Riichiro Saito2 1Frontier Research Institute for
Interdisciplinary Sciences, Tohoku University, Sendai 980-8578, Japan
2Department of Physics, Tohoku University, Sendai 980-8578, Japan
3Department of Physics, Institut Teknologi Bandung, Bandung 40132, Indonesia
4Research Center for Quantum Physics, National Research and Innovation Agency
(BRIN), South Tangerang 15314, Indonesia

###### Abstract

The effects of the unique density of states (DOS) of a topological type-II
nodal-line semimetal (NLS) on its thermoelectric (TE) transport properties are
investigated through a combination of semi-analytical and first-principles
methods with “spinless Mg3Bi2” as artificial material. The DOS in such a type-
II NLS possesses two van Hove singularities near the energy of the nodal line
that leads to a large $S$ value compared to the normal metals. Combined with
the linear band at the nodal line that gives high electrical conductivity
$\sigma$, the type-II NLS can exhibit a relatively high TE power factor
($\mathrm{PF}=S^{2}\sigma$) at the nodal line. In particular, we find
$\text{PF}\sim 60$ $\mathrm{\SIUnitSymbolMicro}$W/cmK2 at 300 K for the n-type
Mg3Bi2 by considering the electron-phonon scattering, in which the relaxation
time $\tau$ of carriers can be expressed as $\tau\propto\text{DOS}^{-1}$ for
the type-II NLS. Furthermore, we optimize parameters for the TE power factor
of type-II NLSs in general by adopting the two-band model with the DOS-
dependent relaxation time approximation. Our results suggest the type-II NLSs
as a potential class of high-performance TE materials among metals and
semimetals, which are traditionally considered not good TE materials compared
to semiconductors.

###### pacs:

72.20.Pa,72.10.-d,73.50.Lw

Thermoelectric (TE) effects were first reported in metals, in which a voltage
difference $\Delta V$ occurs as an electron flows from the hot side to the
cold side when a temperature difference $\Delta T$ is applied Rowe (2005). The
first measure of thermoelectricity is the Seebeck coefficient, $S=-\Delta
V/\Delta T$ Goldsmid (2010). Searching a material with a high $S$ is thus
necessary to develop high-performance TE generators or cooling devices because
$S$ is related to the TE device performance characterized by the figure of
merit $ZT=S^{2}\sigma T/(\kappa_{e}+\kappa_{ph})$ Goldsmid (2010), where
$\sigma$ is the electrical conductivity, $\kappa_{e}$ is the electronic
thermal conductivity, $\kappa_{ph}$ is the lattice thermal conductivity and
$T$ is the absolute temperature. Although metals were studied in the early
stages of thermoelectrics discovery, their $S$ is usually much smaller than
$k_{B}/e\approx 87$ $\mathrm{\SIUnitSymbolMicro}$V/K, where $k_{B}$ is the
Boltzmann constant and $e$ is the electron charge. For example, $|S|$ of
copper is $6.5$ $\mathrm{\SIUnitSymbolMicro}$V/K at $300$ K. To obtain a
better value of $ZT$, people often use doped semiconductors, in which $S$ can
be larger than in the metals, on the order of $\mu/k_{B}T$, where $\mu$ is the
energy difference between the Fermi energy and the top (or bottom) of the
conduction (or valence) band for the n-type (or p-type) material. For example,
$|S|$ of n-type silicon is $340$ $\mathrm{\SIUnitSymbolMicro}$V/K at $300$ K
for doping concentration of $2.2\times 10^{18}$ cm-3 Geballe and Hull (1955).
However, a large $S$ would lead to a small value of $\sigma$ in the
semiconductors Hung  _et al._ (2015). Therefore, finding materials with both
large $S$ and high $\sigma$ is the bottleneck to improve the TE power factor,
$\text{PF}=S^{2}\sigma$. In this regard, many studies have been focusing on
the PF enhancement for the semiconductors through, for example, band
convergence Hung  _et al._ (2019); Tang  _et al._ (2015) and quantum
confinement Hicks and Dresselhaus (1993a, b); Hung  _et al._ (2016); Hung and
Saito (2021). However, the PF enhancement in the metals or semimetals remains
a long-standing problem.

Recently, there has been a lot of attention on materials with non-trivial band
topology such as the nodal line semimetals (NLSs) Shao  _et al._ (2020);
Rudenko  _et al._ (2018), in which the conduction and valence bands intersect
in the form of a line (called the nodal line) Fu  _et al._ (2019). Based on
the slopes of the bands along the nodal line, we can classify the NLSs into
type-I and type-II NLSs. The type-I NLS has bands with oppositely aligned
slopes near the nodal line, while the type-II NLS has bands with the same
slope near the nodal line. Although the semimetals are not suitable for TE
applications due to the opposite contribution of electrons and holes to the
Seebeck coefficient Hasdeo  _et al._ (2019), some recent findings pointed out
that NLSs might be promising TE materials, such as ZrSiS Matusiak  _et al._
(2017), Nb3GeTe6 Wang  _et al._ (2020) or YbMnSb2 Pan  _et al._ (2021). A
previous theoretical study by Zhang et al. suggested that non-trivial
electrons in the semimetal ZrTe5 lead to the anomalous Nernst effect and the
Seebeck effect Zhang  _et al._ (2019), in which both the Seebeck coefficient
and Nernst signal exhibit exotic peaks in the strong-field quantum limit. The
anomalous transport behaviors of ZrTe5 observed by Shahi et al. Shahi  _et
al._ (2018). Xu et al. Xu  _et al._ (2014) also found that the edge states of
the topological semimetal lead to the large and anomalous Seebeck effect with
an opposite sign with respect to the Hall effect.

In this Letter, we investigate the TE response of type-II NLSs using the semi-
analytical method by the two-band model (2BM) and the first-principles
calculation based on density functional theory (DFT). As an example of the
type-II NLSs, we adopt the three-dimensional (3D) Mg3Bi2. As shown in Fig.
1(a), the type-II NLS can be described by a combination of a linear band (LB),
$E_{c}(\bm{k})$, and a Mexican-hat band (MB), $E_{v}(\bm{k})$, defined by

$E_{c}(\bm{k})=\hbar v_{F}|\bm{k}|,$ (1)

and

$E_{v}(\bm{k})=-\frac{(\hbar^{2}\bm{k}^{2}/4m-E_{1})^{2}}{E_{1}}+E_{0},$ (2)

respectively, where $\bm{k}$ is the electron wavevector, $v_{F}$ is the Fermi
velocity in the LB, $E_{0}$ [Fig. 1(b)] is a parameter that determines the
position of the MB edge with respect to $E=0$, $E_{1}$ is a parameter which
determines the depth of the central valley of the MB measured from the band
edge, $m$ is the effective mass of valley in the middle of the MB, and $\hbar$
is the reduced Planck constant.

Figure 1: Type-II nodal-line semimetal (NLS). (a) Two-band model for type-II
NLS with a linear band and a Mexican-hat band. The dashed line denotes the
nodal line. (b) Total density of states (solid line) of the linear band
(green) and Mexican-hat band (red) has two van Hove singularities (vHSs) at
$E_{0}$ and $E_{0}-E_{1}$ [see Eq. (3)]. The dashed lines (with shaded areas)
in the DOS plot represent the individual contribution of each band.

From Eqs. (1) and (2), the electronic density of states (DOS),
$\mathcal{D}_{2b}(E)$, within the two-band model of 3D type-II NLS can be
written as (see the Supplemental Material for the derivation):

$\mathcal{D}_{2b}(E)=\begin{cases}\frac{2m\sqrt{mE_{1}}}{\pi^{2}\hbar^{3}}\sqrt{\frac{1+\sqrt{(E_{0}-E)/E_{1}}}{(E_{0}-E)/E_{1}}}&\quad\text{for
}E\leq 0,\\\
\frac{E^{2}}{\pi^{2}\hbar^{3}v_{F}^{3}}+\frac{2m\sqrt{mE_{1}}}{\pi^{2}\hbar^{3}}\sqrt{\frac{1+\sqrt{(E_{0}-E)/E_{1}}}{(E_{0}-E)/E_{1}}}&\quad\text{for
}0<E<E_{0}-E_{1},\\\
\frac{E^{2}}{\pi^{2}\hbar^{3}v_{F}^{3}}+\frac{2m\sqrt{mE_{1}}}{\pi^{2}\hbar^{3}}\left(\sqrt{\frac{1-\sqrt{(E_{0}-E)/E_{1}}}{(E_{0}-E)/E_{1}}}+\sqrt{\frac{1+\sqrt{(E_{0}-E)/E_{1}}}{(E_{0}-E)/E_{1}}}\right)&\quad\text{for
}E_{0}-E_{1}\leq E<E_{0},\\\
\frac{E^{2}}{\pi^{2}\hbar^{3}v_{F}^{3}}&\quad\text{for }E\geq E_{0}.\\\
\end{cases}$ (3)

Equation (3) shows two van Hove singularities (vHSs): one from the divergence
of the DOS at $E=E_{0}$ and the other from the discontinuity at
$E=E_{0}-E_{1}$, as shown in Fig. 1(b). The nodal line exists at the energy
between the two vHSs. Therefore, the enhancement of DOS occurs in a small
region between the two vHSs for the type-II NLS. The vHSs also enhance the PF
of the low-dimensional materials due to the quantum confinement effect Hicks
and Dresselhaus (1993b); Hung  _et al._ (2016); Hung and Saito (2021). On the
other hand, the LB generally gives a large $\sigma$ as reported in the cases
of graphene Stoller  _et al._ (2008) and three-dimensional Dirac Cd3As2
Neupane  _et al._ (2014).

Figure 2: Mg3Bi2 as an example of type-II NLSs. (a) Crystal structure of
Mg3Bi2. The orange, green, and blue balls indicate the Bi, Mg1, and Mg2 atoms,
respectively. (b) Coordination polyhedrons of the non-equivalent Mg1 and Mg2
atoms. (c) Electronic band structure with atomic orbital projections in color.
The inset shows a tiny gap and a type-II Dirac point along the $\Gamma$-M and
$\Gamma$-K directions, respectively. (d) Electronic density of states (DOS)
and scattering rate at 300 K as a function of energy. A local increase in DOS
occurs near the nodal line at $\sim 0.1$ eV.

In order to show an example of the type-II NLS, we adopt 3D Mg3Bi2 whose
electronic structure and phonon-related properties are calculated with the
first-principles DFT method. We use Quantum ESPRESSO Giannozzi  _et al._
(2009) to perform the DFT calculations with the Perdew-Burke-Ernzerhof
parameterization Perdew  _et al._ (1996) for the exchange-correlation
functional under the generalized gradient approximation. Ultrasoft
pseudopotentials with a kinetic energy cutoff of 60 Ry are also employed in
the calculations. We then use the EPW package Poncé  _et al._ (2016) to
calculate the carrier scattering rate, which is given by the inverse of the
relaxation time $\tau_{m\mathbf{k}}^{\mathrm{el-ph}}$ by the electron-phonon
interaction as

$\begin{split}\frac{1}{\tau_{m\mathbf{k}}^{\mathrm{el-
ph}}}=&\frac{2\pi}{\hbar}\sum_{m^{\prime}\nu
j}\int\frac{d\mathbf{q}}{\Omega_{\text{BZ}}}|\mathcal{M}(\mathbf{k},\mathbf{q})|^{2}\times\\\
&\left[\frac{1+j}{2}-jf_{m^{\prime}\mathbf{k}+\mathbf{q}}^{0}+n_{\mathrm{q}\nu}\right]\delta(\Delta
E_{{\mathbf{k},\mathbf{q}}}-j\hbar\omega_{\mathbf{q}\nu}),\end{split}$ (4)

where $\mathcal{M}(\mathbf{k},\mathbf{q})$ is the electron-phonon matrix
element for the scattering of an electron from the initial state $m\mathbf{k}$
to the final state $m^{\prime}\mathbf{k}+\mathbf{q}$ by emitting (or
absorbing) a $\nu$-th phonon with the wavevector $\mathbf{q}$ and frequency
$\omega_{\mathbf{q}\nu}$. We consider all electron-phonon scattering
processes, including acoustic, optical, and polar-optical phonons for
$\mathcal{M}(\mathbf{k},\mathbf{q})$. In Eq. (4),
$f_{m^{\prime}\mathbf{k}+\mathbf{q}}^{0}$ and $n_{\mathrm{q}\nu}$ are the
Fermi-Dirac and Bose-Einstein distribution functions, respectively. The index
$j$ takes a value of $\pm 1$ for phonon emission ($+$) and absorption ($-$),
respectively. The electron energy and phonon dispersion relations in Eq. (4)
are initially calculated on $10\times 10\times 6$ $\mathbf{k}$-point and
$5\times 5\times 3$ $\mathbf{q}$-point grids, respectively. Then, we
interpolate the electronic energy and phonon dispersions on the dense meshes
of $70\times 70\times 42$ $\mathbf{k}$-point and $35\times 35\times 21$
$\mathbf{q}$-point grids for calculating $\tau_{m\mathbf{k}}^{\mathrm{el-
ph}}$, respectively.

In Fig. 2(a), we show the crystal structure of Mg3Bi2, in which the lattice is
centrosymmetric with space group of $P\bar{3}m1$. The calculated lattice
constants by geometry optimization are $a=b=4.683$ Å and $c=7.396$ Å and
reproduce the experimental ones Watson  _et al._ (1984) ($a=b=4.666$ Å and
$c=7.401$ Å) with the deviation $<0.4$ %. The Mg3Bi2 structure can be
described by the ionic layers of the Mg1 atoms sandwiched by two-dimensional
(2D) covalent layers of the Mg2 and Bi atoms. The Mg1 and Mg2 atoms are non-
equivalent atoms with different coordination, as shown in Fig. 2(b). The Mg1
atom is connected to six Bi atoms with six equal bond lengths of $3.18$ Å,
while the Mg2 atom is tetrahedrally coordinated by the four Bi atoms with
three shorter bond lengths of $2.95$ Å and one vertical bond length of $3.03$
Å. Therefore, the Mg1 and Mg2 atoms contribute differently to the electronic
band structure.

In Fig. 2(c), we show the electronic band structure of Mg3Bi2 with atomic
orbital projections in color. Here we do not consider spin-orbit interaction.
Near the Fermi energy ($E=0.11$ eV) and along the $\Gamma$-M and $\Gamma$-K
directions, two bands characterized by the LB and MB [see Fig. 1(a)] cross
each other, indicating a semimetallic property in Mg3Bi2. The atomic orbital
projections indicate that the LB and MB originate from the Mg2 and Bi atoms of
the 2D covalent layer, respectively. By zooming the band overlap region [inset
of Fig. 2(c)], we find that the crossing point is gapless along the $\Gamma$-K
direction, and the slopes of the two bands at the crossing point have the same
sign, indicating a type-II Dirac point (DP). On the other hand, along the
$\Gamma$-M direction, there is a tiny gap ($\sim 1$ meV) between the two
bands. Quan et al. Quan  _et al._ (2017) demonstrated that in a spinless
system with both space-inversion and time-reversal symmetries, the crossing
points must form a one-dimensional (1D) line (or nodal line) rather than
discrete points. In our calculation, the nodal line around the $\Gamma$ point
contains the type-II DP along the $\Gamma$-K direction. However, the nodal
line does not exactly lie on the $\Gamma$-M-K plane but slightly wiggles
around this plane. Therefore, a tiny gap exists along the $\Gamma$-M
direction. This result is consistent with previous experimental and
theoretical studies Chang  _et al._ (2019); Zhang  _et al._ (2017). It is
important to note that the type-II NLS occurs in Mg3Bi2 only for the spinless
case. When we take into account the spin-orbit interaction in the band
calculation, an indirect gap of 35 meV appears along the $\Gamma$-M direction
(see Fig. S4 in the Supplemental Material). Thus, real Mg3Bi2 is no longer the
type-II NLS. Since the present study focuses on the conventional TE properties
of the type-II NLS, we consider only the spinless Mg3Bi2 case. Our calculated
results do not correspond to real Mg3Bi2 with spin-orbit coupling, yet it can
give insight into desired enhancement of TE properties in the real system.

In Fig. 2(d), we show the DOS (red line) and the scattering rate
($1/\tau^{\mathrm{el-ph}}$) (blue region) at 300 K of the spinless Mg3Bi2 as a
function of the energy. The enhancement of DOS is found at the energy of nodal
line ($\sim 0.1$ eV) due to the vHSs, as discussed by the two-band model. The
enhancement of DOS has the window energy about 0.125 eV, which is much larger
than $k_{B}T=0.025$ eV at the room temperature. Therefore, the electrons can
exist on the enhancement of DOS even for the room temperature. The scattering
rate shows a value of relaxation time $\tau^{\mathrm{el-ph}}\approx 0.03$ ps
at the nodal line. The energy dependence of the scattering rate suggests that
$\tau^{\mathrm{el-ph}}$ can be expressed by the DOS as
$\tau^{\mathrm{D}}=C/\text{DOS}$, where $C=2.2\times 10^{-13}$ is a constant
in units of $\mathrm{sec}\times\text{state}/\text{Ry}/\text{volume}$, which is
obtained by the least-squares fitting in BoltzTraP2 Madsen  _et al._ (2018).

Using calculated band structure and the relaxation time, $S$, $\sigma$, and
$\kappa_{e}$ are obtained within the linearized Boltzmann theory as Hung  _et
al._ (2018)

$S=\frac{1}{qT}\frac{\mathcal{L}_{1}}{\mathcal{L}_{0}},~{}~{}\sigma=q^{2}\mathcal{L}_{0},~{}~{}\kappa_{e}=\frac{1}{T}\left(\mathcal{L}_{2}-\frac{\mathcal{L}_{1}^{2}}{\mathcal{L}_{0}}\right),$
(5)

where the transport integrals $\mathcal{L}_{i}$ with $i=0,1,2$ are defined by

$\mathcal{L}_{i}=\int{v_{x}^{2}\tau(E)\mathcal{D}(E)(E-\mu)^{i}\left(-\frac{\partial
f_{0}}{\partial E}\right)dE},$ (6)

where $v_{x}$ is the group velocity in the $x$ direction. Eqs. (5) and (6) are
calculated by using the BoltzTraP2 package Madsen  _et al._ (2018). We
consider three scattering models for the carrier relaxation-time approximation
(RTA): (1) $\tau(E)$ within the electron-phonon coupling approximation (EPA),
represented by $\tau(E)=\tau^{\mathrm{el-ph}}$, (2) $\tau(E)$ inversely
proportional to the total DOS (DRTA) of all bands, denoted by
$\tau(E)=C/\mathcal{D}$, and (3) the constant RTA (CRTA), represented by
$\tau(E)=\tau_{0}$ with $\tau_{0}$ is set to $3\times 10^{-14}$ s. For further
insight into the relevant scattering mechanisms, we also calculate the TE
properties of Mg3Bi2 semi-analytically by using the two-band model with
$\tau(E)=C_{0}/\mathcal{D}_{2b}$, where $\mathcal{D}_{2b}$ is given in Eq.
(3), and $C_{0}=3.2\times 10^{-14}$ s/Ry/volume . The detailed derivation and
the fitting parameters for the two-band model can be found in the Supplemental
Material.

Figure 3: (a) Seebeck coefficient $S$, (b) electrical conductivity $\sigma$,
(c) thermoelectric power factor PF, and (d) figure of merit $ZT$ of spinless
Mg3Bi2 NLS are plotted as a function of the chemical potential $\mu$ at 300 K.
Four scattering-models are calculated for the relaxation-time approximation,
including the EPA ($\tau(E)=\tau^{\mathrm{el-ph}}$), the DRTA ($C/\mathcal{D}$
and $C_{0}/\mathcal{D}_{2b}$), and CRTA ($\tau_{0}$). Yellow and blue regions
denote the p-type and n-type, respectively. The optimized PF and $ZT$ values
are found at the nodal line $\mu\sim 0.1$ eV for n-type Mg3Bi2.

In Figs. 3(a)-(d), we show $S$, $\sigma$, PF, and $ZT$ for the $x$ direction
of Mg3Bi2 as a function of $\mu$ at $T=300$ K, respectively. In present study,
we focus on the in-plane TE properties since Mg3Bi2 has the layer structure,
as shown in Fig. 2(a). It is noted that $S$ and $\sigma$ in the $y$ direction
are similar with that in the $x$ direction. As shown in Figs. 3(a) and (b),
the DRTA (i.e., $\tau(E)=C/\mathcal{D}$ and $C_{0}/\mathcal{D}_{2b}$) can
reproduce the asymmetry (with respect to p- and n-type regions) of $|S|$ and
$\sigma$ found in the case of the EPA (i.e., $\tau(E)=\tau^{\mathrm{el-ph}}$).
The asymmetry of TE properties can be explained by the term of $v^{2}_{x}$ in
Eq. (6) when the term of $\tau(E)\mathcal{D}(E)$ becomes a constant ($C$ or
$C_{0}$) for the DRTA cases. On the other hand, the CRTA (i.e.,
$\tau(E)=\tau_{0}$) shows almost symmetric $S$ and $\sigma$ with respect to
p-type and n-type regions. This result suggests that the CRTA is not suitable
to describe the TE properties of the type-II NLS even though it is widely used
for TE properties calculation for semiconductor.

The optimized value of $|S|$ is found to be 100
$\mathrm{\SIUnitSymbolMicro}$V/K at $\mu=0.08$ eV for the n-type, as shown in
Fig. 3(a), which is higher than upper limit of the metal ($k_{B}/e\approx 87$
$\mathrm{\SIUnitSymbolMicro}$V/K). For graphene, the maximum $|S|$ is reported
about 60 $\mathrm{\SIUnitSymbolMicro}$V/K for the p-type Kanahashi  _et al._
(2019). The large $|S|$ of Mg3Bi2 at near the nodal line can be explained by
the enhancement of DOS, as shown in Eq. (3). On the other hand, we have
$\sigma\propto\mathcal{L}_{0}\propto v_{x}$ from Eqs. (5) and (6)) within the
EPA and DRTA. Therefore, $\sigma$ for the n-type is larger than that for the
p-type because of the contribution of the LB, as shown in Fig. 3(b).

Based on $S$ and $\sigma$, the optimized PF $\sim 55$
$\mathrm{\SIUnitSymbolMicro}$W/cmK2 is found at $\mu\sim 0.1$ eV for the
n-type Mg3Bi2, which is much larger than the p-type case. This optimized PF is
larger than the experiment value for the n-type Mg3.2Bi2-xSbx at room
temperature ($\sim 25-28$ $\mathrm{\SIUnitSymbolMicro}$W/cmK2 Mao  _et al._
(2019); Liang  _et al._ (2021)). Zhang et al. Zhang and Iversen (2019) also
reported the PF $\sim 14$ $\mathrm{\SIUnitSymbolMicro}$W/cmK2 for the n-type
Mg3Bi2 with spin-orbit coupling within the CRTA. We note that due to the spin-
orbit interaction and the doping effect, Mg3Bi2 shows a transition from the
NLS to metal or semiconductor, which might lead to reducing the PF compared
with the spinless case.

In Fig. 3(d), we show the $ZT=\text{PF}/(\kappa_{e}+\kappa_{ph})$ value as a
function of $\mu$ at $T=300$ K. Here, we adopt $\kappa_{ph}=1$ W/mK from the
experimental data Mao  _et al._ (2019) of Mg3Bi2, and the calculated
$\kappa_{e}$ is around 5 W/mK, as shown in Fig. S5 in the Supplemental
Material. The optimized $ZT\sim 0.3$ is found at $\mu\sim 0.1$ eV for the EPA
case. Although $ZT$ of the NLS is smaller than the TE semiconductor, the high
PF of the NLS is important for the specific TE application. In particular,
when the heat source is unlimited, such as solar heat, the output power
density, $Q\propto\text{PF}$, is an essential parameter to evaluate the
performace and TE generation cost ($ per W) Yee  _et al._ (2013); Liu  _et
al._ (2015).

Using the two-band model, we show the optimized value of the PF as a function
of $E_{1}$ and $E_{0}$ (see Fig. S3 in the Supplemental Material). We suggest
that the best type-II NLS for the TE should have a small $E_{0}-E_{1}$ on the
order of a few meV, but a condition of $E_{1}<E_{0}\neq 0$ should be satisfied
to keep the material as the type-II NLS. We also calculate the TE properties
of Mg3Bi2 with different temperatures (200K, 300K, 400K, and 500K, see Fig. S6
in the Supplemental Material), which shows that the PF increases with
decreasing the temperature. This suggests Mg3Bi2 as a potential n-type TE
material for low-temperature energy harvesting.

In conclusion, we have shown that the enhancement of DOS of type-II NLS leads
to a high Seebeck coefficient at the nodal line region, which can be higher
than the upper limit of the metal $k_{B}/e\approx 87$
$\mathrm{\SIUnitSymbolMicro}$V/K. On the other hand, the linear band of the
type-II NLS also contributes to high electrical conductivity. Therefore, we
can obtain an optimized power factor with the chemical potential at the nodal
line. The unique DOS of the type-II NLS plays an important role in improving
the TE properties of the semimetal.

N.T.H. acknowledges JSPS KAKENHI (Grant No. JP20K15178) and the financial
support from the Frontier Research Institute for Interdisciplinary Sciences,
Tohoku University. J.M.A. and A.R.T.N. performed some of the calculations with
the help of HPC LIPI facility. R.S. acknowledges JSPS KAKENHI Grant No
JP18H01810.

## References

  * Rowe (2005) D. M. Rowe,  _Thermoelectrics handbook: macro to nano_ (CRC press, Boca Ratcon, 2005).
  * Goldsmid (2010) H. J. Goldsmid,  _Introduction to Thermoelectricity_ (Springer-Verlag, Berlin Heidelberg, 2010).
  * Geballe and Hull (1955) T. H. Geballe and G. W. Hull, Phys. Rev. 98, 940 (1955).
  * Hung  _et al._ (2015) N. T. Hung, A. R. T. Nugraha, E. H. Hasdeo, M. S. Dresselhaus, and R. Saito, Phys. Rev. B 92, 165426 (2015).
  * Hung  _et al._ (2019) N. T. Hung, A. R. T. Nugraha, T. Yang, Z. Zhang, and R. Saito, J. Appl. Phys. 125, 082502 (2019).
  * Tang  _et al._ (2015) Y. Tang, Z. M. Gibbs, L. A. Agapito, G. Li, H. S. Kim, M. B. Nardelli, S. Curtarolo, and G. J. Snyder, Nat. Mater. 14, 1223 (2015).
  * Hicks and Dresselhaus (1993a) L. D. Hicks and M. S. Dresselhaus, Phys. Rev. B 47, 16631 (1993a).
  * Hicks and Dresselhaus (1993b) L. D. Hicks and M. S. Dresselhaus, Phys. Rev. B 47, 12727 (1993b).
  * Hung  _et al._ (2016) N. T. Hung, E. H. Hasdeo, A. R. T. Nugraha, M. S. Dresselhaus, and R. Saito, Phys. Rev. Lett. 117, 036602 (2016).
  * Hung and Saito (2021) N. T. Hung and R. Saito, Adv. Quantum Technol. 4, 2000115 (2021).
  * Shao  _et al._ (2020) Y. Shao, A. N. Rudenko, J. Hu, Z. Sun, Y. Zhu, S. Moon, A. Millis, S. Y., A. I. Lichtenstein, D. Smirnov, Z. Q. Mao, M. I. Katsnelson, and D. N. Basov, Nat. Phys. 16, 636 (2020).
  * Rudenko  _et al._ (2018) A. N. Rudenko, E. A. Stepanov, A. I. Lichtenstein, and M. I. Katsnelson, Phys. Rev. Lett. 120, 216401 (2018).
  * Fu  _et al._ (2019) B. B. Fu, C. J. Yi, T. T. Zhang, M. Caputo, J. Z. Ma, X. Gao, B. Q. Lv, L. Y. Kong, Y. B. Huang, P. Richard, M. Shi, V. N. Strocov, H. M. Weng, Y. G. Shi, T. Qian, and H. Ding, Science advances 5, eaau6459 (2019).
  * Hasdeo  _et al._ (2019) E. H. Hasdeo, L. P. A. Krisna, M. Y. Hanna, B. E. Gunara, N. T. Hung, and A. R. T. Nugraha, J. Appl. Phys. 126, 035109 (2019).
  * Matusiak  _et al._ (2017) M. Matusiak, J. R. Cooper, and D. Kaczorowski, Nat. Commun. 8, 1 (2017).
  * Wang  _et al._ (2020) X. Wang, G. Ding, S. A. Khandy, Z. Cheng, G. Zhang, X. L. Wang, and H. Chen, Nanoscale 12, 16910 (2020).
  * Pan  _et al._ (2021) Y. Pan, F. R. Fan, X. Hong, B. He, C. Le, W. Schnelle, Y. He, K. Imasato, H. Borrmann, C. Hess, B. Buchner, Y. Sun, C. Fu, G. J. Snyder, and C. Felser, Adv. Mater. 33, 2003168 (2021).
  * Zhang  _et al._ (2019) J. L. Zhang, C. M. Wang, C. Y. Guo, X. D. Zhu, Y. Zhang, J. Y. Yang, Y. Q. Wang, Z. Qu, L. Pi, H. Z. Lu, and M. L. Tian, Phys. Rev. Lett. 123, 196602 (2019).
  * Shahi  _et al._ (2018) P. Shahi, D. J. Singh, J. P. Sun, L. X. Zhao, G. F. Chen, Y. Y. Lv, J. Li, J. Q. Yan, D. G. Mandrus, and J. G. Cheng, Phys. Rev. X 8, 021055 (2018).
  * Xu  _et al._ (2014) Y. Xu, Z. Gan, and S. C. Zhang, Phys. Rev. Lett. 112, 226801 (2014).
  * Stoller  _et al._ (2008) M. D. Stoller, S. Park, Y. Zhu, J. An, and R. S. Ruoff, Nano Lett. 8, 3498 (2008).
  * Neupane  _et al._ (2014) M. Neupane, S. Y. Xu, R. Sankar, N. Alidoust, G. Bian, C. Liu, I. Belopolski, T. R. Chang, H. T. Jeng, H. Lin, A. Bansil, F. Chou, and M. Z. Hasan, Nat. Commun. 5, 1 (2014).
  * Giannozzi  _et al._ (2009) P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. D. Corso, S. de Gironcoli, S. Fabris, G. Fratesi, R. Gebauer, U. Gerstmann, C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari, F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto, C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov, P. Umari, and R. M. Wentzcovitch, J. Phys. Condens. Matter 21, 395502 (2009).
  * Perdew  _et al._ (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * Poncé  _et al._ (2016) S. Poncé, E. R. Margine, C. Verdi, and F. Giustino, Comput. Phys. Commun. 209, 116 (2016).
  * Watson  _et al._ (1984) L. M. Watson, C. A. W. Marshall, and C. P. Cardoso, J. Phys. F: Metal Phys. 14, 113 (1984).
  * Quan  _et al._ (2017) Y. Quan, Z. P. Yin, and W. E. Pickett, Phys. Rev. Lett. 118, 176402 (2017).
  * Chang  _et al._ (2019) T. R. Chang, I. Pletikosic, T. Kong, G. Bian, A. Huang, J. Denlinger, S. K. Kushwaha, B. Sinkovic, H. T. Jeng, T. Valla, W. Xie, and R. Cava, Adv. Sci. 6, 1800897 (2019).
  * Zhang  _et al._ (2017) X. Zhang, L. Jin, X. Dai, and G. Liu, J. Phys. Chem. Lett. 8, 4814 (2017).
  * Madsen  _et al._ (2018) G. K. H. Madsen, J. Carrete, and M. J. Verstraete, Comp. Phys. Commun. 231, 140 (2018).
  * Hung  _et al._ (2018) N. T. Hung, A. R. T. Nugraha, and R. Saito, Phys. Rev. Applied 9, 024019 (2018).
  * Kanahashi  _et al._ (2019) K. Kanahashi, M. Ishihara, M. Hasegawa, H. Ohta, and T. Takenobu, npj 2D Mater. Appl. 3, 1 (2019).
  * Mao  _et al._ (2019) J. Mao, H. Zhu, Z. Ding, Z. Liu, G. A. Gamage, G. Chen, and Z. Ren, Science 365, 495 (2019).
  * Liang  _et al._ (2021) Z. Liang, C. Xu, H. Shang, Q. Zhu, F. Ding, J. Mao, and Z. Ren, Mater. Today Phys. 19, 100413 (2021).
  * Zhang and Iversen (2019) J. Zhang and B. B. Iversen, J. Appl. Phys. 126, 085104 (2019).
  * Yee  _et al._ (2013) S. K. Yee, S. LeBlanc, K. E. Goodson, and C. Dames, Energy Environ. Sci. 6, 2561 (2013).
  * Liu  _et al._ (2015) W. Liu, H. S. Kim, S. Chen, Q. Jie, B. Lv, M. Yao, Z. Ren, C. P. Opeil, S. Wilson, C. W. Chu, and Z. Ren, Proc. Natl. Acad. Sci. U.S.A 112, 3269 (2015).