# BLDNet: A Semi-supervised Change Detection Building Damage Framework using
Graph Convolutional Networks and Urban Domain Knowledge ††thanks: Citation:
Authors. Title. Pages…. DOI:000000/11111.

Ali Ismail
Department of Electrical and Computer Engineering
American University of Beirut
Beirut, Lebanon
<EMAIL_ADDRESS>
Mariette Awad
Department of Electrical and Computer Engineering
American University of Beirut
Beirut, Lebanon
<EMAIL_ADDRESS>
0000-0001-8636-1373 0000-0002-4815-6894

###### Abstract

Change detection is instrumental to localize damage and understand destruction
in disaster informatics. While convolutional neural networks are at the core
of recent change detection solutions, we present in this work, BLDNet, a novel
graph formulation for building damage change detection and enable learning
relationships and representations from both local patterns and non-stationary
neighborhoods. More specifically, we use graph convolutional networks to
efficiently learn these features in a semi-supervised framework with few
annotated data. Additionally, BLDNet formulation allows for the injection of
additional contextual building meta-features. We train and benchmark on the
xBD dataset to validate the effectiveness of our approach. We also demonstrate
on urban data from the 2020 Beirut Port Explosion that performance is improved
by incorporating domain knowledge building meta-features.

_K_ eywords Change detection $\cdot$ Building damage $\cdot$ Urban data
science $\cdot$ Disaster informatics $\cdot$ Graph convolutional networks
$\cdot$ Semi-supervised learning $\cdot$ Meta-features $\cdot$ Siamese
networks $\cdot$ Domain knowledge

##  1 Introduction

Change detection (CD) is a sub-field of remote sensing concerned with
identifying and localizing differences in surface objects using images taken
at different times. CD has been successfully applied in many different fields
such as forest monitoring [1, 2, 3], surface water monitoring [4, 5], sea ice
monitoring [6, 7], landslide monitoring [8], tsunamis [9, 10], fires [11],
urban land use [12, 13, 14] and planetary surfaces [15]. In disaster
informatics such as the world trade center attacks and hurricane Katrina, [16]
building damage assessment was mostly emphasized [17].

Traditionally, CD was performed using classical image processing techniques
[18]. With the increasing availability of huge amounts of data and with the AI
revolution, new CD techniques based on machine learning have emerged [19].
CNNs are currently the most used architecture for CD [20]. Various CNN
architectures have been used for CD such as object detection [1] and semantic
segmentation [21]. Since most CD applications use pre and post images, Siamese
CNN architectures are particularly popular [22, 23]. Due to the temporal
aspect of CD, recurrent neural networks have also been used where the pre
image is considered to be the first timestep followed by the post image [24].
These models have been subsequently merged with CNNs and Siamese CNNs to
exploit their powerful image representation abilities [25, 26, 27]. More
specifically, CD for building damage has been addressed using similar
approaches [28, 29, 30, 31, 32, 33, 10].

However, CNNs can only extract spatial information within the defined
neighborhood of the convolution kernel. Graph convolutional networks (GCN)
instead model data as graphs [34] and explore non-euclidean relationships that
go beyond the pixel neighborhood defined by a CNN kernel [35]. GCNs have
outperformed CNN-based models in many computer vision applications [36, 35]
including urban change detection [37].

On disaster onset, a lot of data is available almost immediately [16]. While
such data will enable training connectionist models, labeling is a major
challenge for such humanitarian applications. Besides, many cities include a
mixture of buildings of various shapes, heights, time periods, building
material and architectural styles. Usually, these buildings compete for the
urban footprint and are in close proximity to each other. These architectural
and structural differences can cause buildings to interact differently with
the disaster and therefore sustain damage to varying levels. While current
approaches rely strictly on satellite or aerial imagery, it may not be
possible to predict well the damage level without the aforementioned
architectural differences embedded in the urban data.

Based on these gaps found in state-of-the-art research, we cast building
damage CD as a node classification problem. This formulation leverages both
the local and neighborhood features and patterns in the data. We present
BLDNet, a hybrid CNN and GCN novel architecture that is trained in a semi-
supervised manner to obtain predictions in a timely manner. Furthermore, we
demonstrate how architectural and contextual building features which we
henceforth call meta-features can be incorporated as domain knowledge into the
graph to improve the prediction results.

The contributions of this paper are:

  * •

A novel formulation of building damage identification as a graph node
classification to learn representations based on local features as well as
relationships with neighboring samples.

  * •

BLDNet, a novel architecture based on a Siamese CNN combined with a GCN
trained in a semi-supervised manner to reduce the number of labeled samples
needed to obtain new predictions.

  * •

A case study on the urban data collected from Beirut Port 2020 explosion with
knowledge domain injection.

To the best of our knowledge, there is no prior work which fuses both images
and meta-features in urban data for CD of building damage. Also, there is no
prior work which adopts a meta-feature framework with GCN.

The rest of this document is organized as follows. In Section 2, we review
related CD work and similar GCN applications. In Section 3, we explain BLDNet
methodology. Section 4 details the datasets used, experiments performed, and
results interpretation while Section 5 concludes the work with future research
directions.

##  2 Related Work

In this section we review deep learning approaches for building damage
detection as well as approaches for accelerating the process of obtaining new
detections. For a more comprehensive review of CD and AI, the reader is
referred to the surveys by Shi et al. [19] and Khelifi and Mignotte [20].

###  2.1 AI for Building Damage Change Detection

The objective of CD in building damage is to localize damage and assess its
severity. An object detection problem using Faster Region-based CNN [38], a
classification using single stream Siamese CNNs [31, 33] and a semantic
segmentation using pyramid encoder-decoders [30] which often have Siamese
backbones [39] were proposed. Because building damage data suffers from severe
class imbalance, some works adopted weighted loss functions [40, 33] or
generated positive samples using pixel level reduction and shifting [28] or
generative adversarial networks [40].

###  2.2 Methodologies for Fast Change Detection

To accelerate damage detection on new disasters, literature relies either on
transfer from one disaster/region to another which allows inference without
training or on semi-supervised learning which enables training a model with
fewer labeled samples.

####  2.2.1 Cross-Domain Transfer for Disaster Damage Detection

The works [29, 41] concerned with developing methodologies to infer new
disasters used model architectures similar to the ones described in the
previous paragraph. Other studies extrapolated to different disasters types
[32, 42, 43, 44, 45]. In many of these cases, transferability is dependent on
the similarity between to the train and test regions [29] and is usually
degraded by data heterogeneity [29, 44]. Other solutions proposed including a
small number of test samples with the training data [41, 44], a multi-domain
adaptive batch normalization and a stochastic weight averaging [42].

####  2.2.2 Semi-supervised Learning for Change Detection

Pati et al. automatically labeled data using clustering and fuzzy logic and
used the labels to train a supervised model [46]. Other works detected
building damage with an outlier detection deep autoencoder [10] and an anomaly
detecting generative adversarial network [47]. Peng et al. proposed a semi-
supervised generative adversarial network where the generator is a supervised
segmentation network. The discriminator was trained to distinguish between the
generator’s prediction on unlabeled samples and the ground truth of labeled
samples to improve the unlabeled predictions [48].

###  2.3 Graph Convolutional Networks in Computer Vision

The representation of images as graphs enabled GCNs to solve many computer
vision problems. In the context of remote sensing image retrieval, a Siamese
GCN was built using pre-computed image segment features as nodes. Adjacent
nodes were connected with the edge weight being the pairwise centroid pixel
and segment orientation difference [36, 49]. To improve multilabel image
classification, Chen et al. mapped the logical relationships between the
different classes in a directed graph and trained a stack of GCNs combined
with the output of a CNN [50]. [51] exploited the semi-supervised nature of
GCNs and the ability to learn from image-wide relationships using non-local
graph representation for land use classification. They built their graph using
each pixel as a node and the similarity between every pair of nodes was used
to build the edges to indicate the likelihood of two nodes belonging to the
same class. Hong et al. similarly built a supervised GCN variant trained using
mini-batches for hyperspectral image classification and tested it combined
with a GCN on different output fusion schemes [35].

###  2.4 State-of-the-Art Work

In recent years, deep learning has been the most adopted approach for CD,
owing to its success in computer vision. We show in Figure 1 a timeline for CD
approaches over the years. Classical and shallow learning approaches have seen
a decrease in use since 2019. This was accompanied by a sustained sharp
increase in CNN adoption. Moreover, additional architectures have appeared in
2020 and 2021 such as hybrid GAN-based models as well as GCN.

Figure 1: Evolution of CD methods over the years.

Differently from existing similar works, we formulate a graph node
classification framework for building disaster damage detection. In [37], the
graph nodes were constructed using pre and post superpixel parcels to compute
summary node features. In our work, we learn deep pixel feature maps as node
features instead of calculating summary features as it is done in traditional
CNN frameworks. For graph edges, Saha et al. [37] connected only parcels with
an adjacent pixel while we use a geometric triangulation to ensure that every
building is connected to all of its surrounding neighbors. Furthermore, we
embed meta-features into our graph to improve learning based on domain
knowledge which is different from aforementioned works that relied on image
data only. In terms of accelerating damage assessment for new events, the bulk
of prior art focuses on pre-trained models for direct inference. However, the
generalization of these models is effective to varying degrees. It often
hinges on the similarity between the test and train data and degrades with
data heterogeneity [44]. Many works even included a few labeled samples from
the test set with the training set which makes the process analogous to semi-
supervised learning [41, 44]. Because semi-supervised models are easier to
develop, we capitalized on semi-supervised learning for efficient deployment.
Table 1 highlights the differences between our work and most related prior
art.

Table 1: Qualitative Comparison with the State-of-the-art.

Work | Building Damage | Uses GCN | Semi-supervised | Domain Transfer | Uses Meta-features
---|---|---|---|---|---
[44] | Yes | No | No | Yes | No
[37] | No | Yes | Yes | No | No
This Work | Yes | Yes | Yes | No | Yes

##  3 Methodology

In this section, we present details about the BLDNet formulation and its
training configurations.

###  3.1 Graph Data Formulation

BLDNet is a graph-based approach that exploits both local image features as
well as relationships with neighboring samples. It assumes the existence of
building footprint polygons which are available in the xBD dataset and the
Beirut Dataset from the Beirut Recovery
Map111https://openmaplebanon.org/beirut-recovery-map. In case these footprints
were not available, they can be obtained using a building footprint detector
[33].

Our formulation builds an undirected acyclic graph using the building
polygons. Each node represents one building. The node features are the
concatenation of the pre and post image crops defined by the rectangular
envelope of the building polygons. The crops are resized to a width and height
of $128\times 128$ in order to unify their size and reduce the memory
footprint of the graph. The resulting feature vector is of size $N=128\times
128\times 3\times 2=98304$. To exploit domain knowledge information, we embed
the meta-features at the node level which increases the dimension of the
vector to $N=98324$. The edges are constructed using a Delaunay triangulation
[52] based on the UTM (Universal Transverse Mercator) building envelope
centroid coordinates. Each edge is given a weight measuring the similarity
between the connected nodes according to the equation used by Saha et al. [37]
to build their adjacency matrix. Figure 2 shows a conceptual sketch of the
proposed graph formulation while Figure 3 shows a realized implementation of a
subgraph from the xBD Joplin tornado.

Figure 2: Concept of the Graph Formulation. Different node colors indicate
nodes’ different classes. Figure 3: Subgraph implemented on a sample from the
Joplin Tornado set.

The semi-supervised GCN requires the entirety of the data during training to
be built into a single graph. Only a relatively small number of nodes is
labeled during training and the model learns to propagate labels along the
graph edges to the rest of nodes. We thus constructed one large graph out of
each selected region.

###  3.2 Model Architecture

Using our graph formulation, we employ graph convolutions to aggregate
information from neighboring nodes. However, given that preliminary
experiments have also shown that it is important to have a reliable way of
extracting local features, we developed BLDNet, a hybrid architecture
combining a CNN and a GCN. The CNN backbone is used to extract local image
features and the GCN operator is used to aggregate them with CNN features of
connected nodes. The CNN backbone is a Siamese ResNet34 network with the
classification layers removed [53]. The output of the two-stream network is
the difference between the individual stream outputs which is piped into a GCN
[34] whose output is the classification. The ReLu activation function is used
for all layers except the last layer which uses a Softmax activation. The
ResNet weights are initialized with the ImageNet weights and trained along
with the entire model.

The meta-features are concatenated with the output of the ResNet backbone. The
resulting vector is then inputted through the GCN layers. While this meta-
feature injection method was adopted in other works, it was for different
application areas than urban damage assessment [54, 55]. The diagram in Figure
4 shows the overall layout of BLDNet.

Figure 4: Architecture of BLDNet. Objects in dotted lines indicate meta-
feature injection if these features are available.

##  4 Experiments and Results

In this section, we present the data that was used along with the data
preprocessing steps. We describe the experiments and procedures to implement
and evaluate BLDNet. We also analyze and discuss our results.

###  4.1 Datasets

The proposed approach was tested on both a public dataset to position it with
respect to the literature as well as the Beirut Port Explosion dataset to
demonstrate the effectiveness of our novel meta-feature injection approach.

####  4.1.1 xBD Dataset

The xBD dataset has different cases of city damage for different disaster
types [17]. It contains a collection of manually annotated pre and post
Worldview 2 images. The dataset is organized into two big sets. The Tier 1 set
contains a set of disasters subdivided into train, test and hold partitions.
The Tier 3 set introduces additional disasters.

xBD is the most diverse in terms of regions, disaster types, buildings and
urban density. It contains image chips of earthquake, tsunami, flood, volcano,
wildfire and tornado/hurricane disasters across sixteen regions. The landscape
in these different regions varies from urban scenes such as Mexico to largely
rural regions such as Guatemala.

The dataset is annotated according to a four-class joint damage scale: no
damage, minor damage, major damage and destroyed. It suffers from severe class
imbalance as the majority of buildings are non-damaged. On average, $80.4\%$
of individual buildings across the Tier 3 regions are non-damaged (Figure 5).
Also, since each region is divided into equal image chips, we investigated the
number of chips with at least one damaged instance and found that about $50\%$
on average do not contain any damaged buildings (Figure 6). These insights
helped shape our data sampling strategy concerning which samples can be
discarded.

Figure 5: xBD Tier 3 number of buildings per class. Figure 6: xBD Tier 3 image
chips with at least one damaged building.

####  4.1.2 Beirut Explosion Dataset

Beirut as a city, is an architectural mosaic with irregular and heterogeneous
building patterns with many different layers of architectural styles gradually
added over decades. The architecture spans many centuries and civilizations as
it is home to Roman heritage sites and civil war (1975-1990) ruins. However,
these structures are not located far from luxurious commercial and financial
districts and modern residential high-rises. The architectural styles that dot
the city are also varied and include Ottoman, European and Gothic patterns
[56].

The unfortunate Beirut port explosion occurred on August 4, 2020 resulting in
the death of nearly 135 people, around 5000 injuries and causing severe
structural damage to the surrounding area leaving more than 300000 people
homeless [57]. The strength of the blast was estimated to be equivalent to the
twentieth of the Hiroshima bomb and was listed as the strongest non-nuclear
explosion in the 21st century [58]. Worldview 2 satellite products covering a
large portion of Beirut were donated by Maxar Technologies. The first view was
taken on July 31 2020 and the second post disaster on August 5 2020. The
damage classification was provided by the Open Map Lebanon
team222https://openmaplebanon.org/beirut-recovery-map. These annotations use a
four class damage scale: minor, moderate, major and severe damage.
Furthermore, the heterogeneous and irregular urban and architectural landscape
of Beirut motivated us to use contextual meta-features. These enable our model
to learn the patterns and interactions resulting from buildings’ non-
uniformity. This dataset was obtained from Krayem et al. [59].

The two satellite products (pre and post) were used to produce pansharpened
color images. The building polygons were overlaid on top of the images which
were georeferenced to fit the polygons. A subset of buildings was selected due
to memory constraints. A view of this area as well as the class distribution
of the included polygons are shown in Figure 7 and Figure 8 respectively.

Figure 7: Overview of the selected area. Figure 8: Class distribution of the
selected samples.

Finally, we created a rectangular envelope around each polygon buffered by
five meters in each direction (Figure 9) because the polygons were drawn to
the shape of the building rooftops. For tall structures, this would have
resulted in the cropped image showing only a small portion of the visible part
of the building. Buffering the polygons also helps with mitigating the effect
of georeferencing error. Other works encountered such difficulties and adopted
similar solutions [45, 32].

Figure 9: Buffered mask (red) around the original polygon (orange).

The meta-features dataset contains multiple building features collected from
multiple sources. We retained features related to the number of apartments,
area, perimeter, number of floors, digital surface model (DSM), building
height, construction year, construction era, heritage status and building
function. Despite consolidating the different columns containing the same
feature coming from different sources to minimize missing values, the data
still had a lot of null entries such as the value "Other" in Figure 10. This
Figure shows the distributions of the Heritage and Building Function
attributes while Table 2 reports on some statistics including the number of
missing values for each attribute. Finally, all columns were normalized to a
range between zero and one, with zero representing missing values.

Table 2: Summary Statistics of the Beirut Meta-data.

| Number of apartments | Mean DSM | Mean building height | Area | Perimeter | Era | Built year | Floors
---|---|---|---|---|---|---|---|---
Count | 535 | 1023 | 1023 | 1257 | 1257 | 711 | 789 | 870
Number of nulls | 831 | 343 | 343 | 109 | 109 | 655 | 577 | 496
Mean | 6.72 | 41.24 | 18.47 | 568.03 | 86.95 | 2.74 | 1954.41 | 4.17
Std | 6.95 | 18.39 | 10.88 | 1484.34 | 76.44 | 0.97 | 36.93 | 2.63
Min | 1 | 3.78 | 1.66 | 17.58 | 17.81 | 1 | 1219 | 1
Max | 60 | 113.70 | 95.65 | 16574.86 | 1062.72 | 5 | 2021 | 26

Figure 10: Counts of Beirut heritage buildings and Beirut buildings by
function.

###  4.2 Experimental Setup

All of the experiments were implemented using Python 3.8, Pytorch 1.7.1 and
Pytorch Geometric 1.7.0 [60]. The code base that was used to perform these
experiments is available333https://gitlab.com/awadailab/gcn-remote-sensing.
The Adam optimizer [61] was used with the categorical crossentropy loss
function which included class weights to mitigate the impact of class
imbalance [33]. Additionally, the "major damage" and "destroyed" classes in
the xBD dataset were merged into a single class because they are hard to
distinguish. This aggregation was adopted by Xu et al. on their dataset [41].
For similar reasons, we merge the "minor" and "moderate" classes in the Beirut
dataset. For model evaluation and comparison, we compute the accuracy,
precision, recall, specificity and the F1 score.

The number of neurons, number of layers, dropout rate and learning rate were
tuned to reduce overfitting and improve performance on the test set. The final
adopted configuration is two graph convolutional layers with 32 neurons and a
dropout rate of 0.5. The learning rate was set empirically to 0.0003. All of
our BLDNet experiments that included preprocessing the data, building the
graph, training and producing predictions took around 10 to 15 minutes to
complete on a virtual machine running on 8 cores of an AMD EPYC 7551 32-Core
Processor with an Nvidia V100 32GB GPU.

###  4.3 Semi-supervised GCN with the xBD dataset

To evaluate the merit of BLDNet we train and test on each of the Pinery
Bushfire, Joplin Tornado and Nepal Flooding events from the xBD Tier 3 set.
This experiment requires all of the data during training because it is semi-
supervised. Moreover, in the case of GCN, all of the data needs to be loaded
at once into memory to build the full graph.

To efficiently manage memory resources, we pruned the unlabeled samples. As
previously mentioned, each disaster event is divided into equal image chips
(Figure 11). We have also shown (Figure 6) that many chips do not have any
damaged samples at all because the damage is more concentrated around the
damaging force.

Figure 11: Image chips of Joplin Tornado as squares of different color.

Therefore, we discarded image chips that do not have any damaged samples. If
the data still needed to be reduced, a random subset is sampled in way to
preserve the class distribution of the original data.

The obtained samples were fed as input to build the graph. Each node in the
graph is marked as being either training, testing or hold. This assignment is
done randomly while maintaining the same class distribution. The training
nodes are a small number of nodes for which the loss function will be computed
during training to optimize the model. Performance metrics were computed on
testing samples after every epoch for model selection. Once training, the
metrics are recorded on the hold set as well as the entire graph.

We experimented with different sizes for the training set and found that using
20% of the data to be the best compromise in terms of performance and number
of training samples (Figure 12).

Figure 12: Model performance with respect to the training data size.

####  4.3.1 Benchmarking with Multiresolution Autoencoder

To benchmark our model, we trained the semi-supervised Multiresolution
Autoencoder [62] on the same data that was used to train BLDNet with exactly
the same train, test and hold splits. This model was also used for
benchmarking by Saha et al. [37]. Table 3 shows the results for both BLDNet
and the multiresolution autoencoder.

Table 3: Comparison between BLDNet and the Multiresolution Autoencoder (AE)
with bold italics indicating better performance.

Set | AE | BLDNet
---|---|---
Pinery Bushfire |  | Acc | Precision | Recall | Specificity | F1 |  | Acc | Precision | Recall | Specificity | F1
Train | 0.6044 | 0.4935 | 0.4811 | 0.7386 | 0.4495 | Train | 0.9524 | 0.8444 | 0.9813 | 0.9822 | 0.9023
Test | 0.5000 | 0.3338 | 0.3288 | 0.6357 | 0.3013 | Test | 0.7146 | 0.4515 | 0.5057 | 0.7791 | 0.4660
Hold | 0.4773 | 0.3439 | 0.3090 | 0.6651 | 0.3044 | Hold | 0.7145 | 0.4517 | 0.4941 | 0.7743 | 0.4646
Full | 0.3909 | 0.2993 | 0.3226 | 0.6028 | 0.2588 | Full | 0.7247 | 0.4671 | 0.5182 | 0.7918 | 0.4826
Joplin Tornado |  | Acc | Precision | Recall | Specificity | F1 |  | Acc | Precision | Recall | Specificity | F1
Train | 0.6337 | 0.5450 | 0.4925 | 0.8033 | 0.5067 | Train | 0.9634 | 0.9462 | 0.9696 | 0.9839 | 0.6562
Test | 0.5171 | 0.5188 | 0.4274 | 0.7484 | 0.4457 | Test | 0.7610 | 0.6968 | 0.6968 | 0.8802 | 0.6900
Hold | 0.6179 | 0.5382 | 0.4967 | 0.7837 | 0.4967 | Hold | 0.7540 | 0.6987 | 0.6992 | 0.8778 | 0.6914
Full | 0.5988 | 0.5400 | 0.4893 | 0.7809 | 0.4961 | Full | 0.7606 | 0.7049 | 0.7027 | 0.8810 | 0.6954
Nepal Flooding |  | Acc | Precision | Recall | Specificity | F1 |  | Acc | Precision | Recall | Specificity | F1
Train | 0.6484 | 0.3976 | 0.6864 | 0.6836 | 0.3905 | Train | 0.9707 | 0.9310 | 0.9871 | 0.9889 | 0.9566
Test | 0.5463 | 0.3664 | 0.3701 | 0.6958 | 0.3643 | Test | 0.6780 | 0.4907 | 0.4535 | 0.7344 | 0.4623
Hold | 0.5930 | 0.4056 | 0.4121 | 0.7059 | 0.4082 | Hold | 0.7086 | 0.5589 | 0.5199 | 0.7613 | 0.5328
Full | 0.5754 | 0.3712 | 0.3733 | 0.6907 | 0.3720 | Full | 0.7152 | 0.5585 | 0.5244 | 0.7710 | 0.5363

BLDNet systematically outperforms the autoencoder. Figure 13 shows the
difference between the BLDNet and autoencoder hold scores for all three
disasters. We achieved an average increase of $16.3\%$, $14.05\%$, $16.51\%$,
$8.62\%$ and $15.98\%$ for accuracy, precision, recall, specificity and F1
score respectively on the hold set across disasters. Recall saw the highest
average improvement which signifies a notable amelioration in the ability to
detect the positive class (damage). We also see a trend where specificity
tends to be higher than other metrics with a mean difference of $17.95\%$ with
the recall. This measure represents the model performance on the negative
class as opposed to recall. The difference is more pronounced in highly
unbalanced datasets. We computed the Shannon equitability index [63] (range 0
to 1 with 1 being balanced and 0 unbalanced) for each of the sets we used in
both their original and pruned versions. We also calculated the difference
between the specificity and recall scores on the full set for each disaster
(Table 4). First, all pruned sets have a higher index than their original
counterparts which means that our data pruning had a positive impact on the
unbalanced state of the dataset. Also, the difference between specificity and
recall is consistently reduced with lower class imbalance (higher Shannon
index). We therefore conclude that decreasing class imbalance and the
prevalence of non-damaged samples increased the model’s ability to accurately
detect damage.

Figure 13: Performance difference with respect to the autoencoder for all
three disasters. Table 4: Shannon Equitability Index for the Chosen xBD Sets
and the Difference between Specificity and Recall.

| Joplin Tornado | Nepal Flooding | Pinery Bushfire
---|---|---|---
Shannon equitability original | 0.813 | 0.558 | 0.247
Shannon equitability pruned | 0.878 | 0.597 | 0.495
Specificity - Recall | 0.1783 | 0.2466 | 0.2736

Additionally, we investigate the class separation ability of our model by
producing a TSNE (t-distributed stochastic neighbor embedding) [64]
visualization of the node embeddings produced by BLDNet (Figure 14). We notice
that each class forms a cluster. However, the "minor damage" class lies
between the two other classes and does not form a cluster as separated and as
condensed as the other two. We see "minor damage" points distributed within
the other class clusters. This gives credit to our and Xu et al.’s assumption
that intermediate damage classes are ambiguous and harder to distinguish from
other classes [41]. These results are achieved without using an ordinal cross
entropy loss function and therefore the model had no incentive to treat the
classes as being ordered.

Figure 14: BLDNet node embeddings for Joplin tornado.

To verify that the difference in performance between the autoencoder and our
model is not situational, we ran the Paired Student t-test [65] and Wilcoxon
signed rank test [66]. Each of the two models is trained thirty times on a
differently sampled training set and its performance is reported on a hold
set. To ensure that the independence condition is not violated and that there
is no overlap between the re-sampled training sets and the hold set, the
latter is separately partitioned and the remainder of the data is randomly
sampled into a different training set every run. The variable considered for
the test is the difference between the two models’ metrics. A separate test is
carried for each of the calculated metrics with a significance level of 5%.
The null hypothesis was rejected for every metric, which means that the
difference in performance is not due to chance. Looking at the distributions
of the metric populations (Figure 15), it can be argued that the difference is
clearly significant without the need to run a statistical test.

Figure 15: Distribution of the performance metrics obtained after 30 runs with
different data samples.

###  4.4 Semi-supervised GCN with the Beirut Explosion dataset

This experiment demonstrates the effectiveness of our method on a different
scenario while including meta-features.

We run the experiment with the same model and training configurations that
were adopted for the xBD dataset. No data pruning strategy was needed since
the area of interest was selected during preprocessing (Figure 7). The first
experiment was performed without any meta-features and the second with urban
domain knowledge injection. Table 5 shows the results of both experiments.

Table 5: Comparison between BLDNet with or without Meta-features on the Beirut
Data, with bold italics for best metrics.

Beirut |  | Accuracy | Precision | Recall | Specificity | F1
---|---|---|---|---|---|---
Train | 0.9534 | 0.8474 | 0.9804 | 0.9817 | 0.9033
Test | 0.7415 | 0.5930 | 0.6744 | 0.7810 | 0.6189
Hold | 0.7291 | 0.5993 | 0.6826 | 0.7815 | 0.6274
Full | 0.7460 | 0.6241 | 0.7276 | 0.8031 | 0.6595
Beirut Meta-features |  | Accuracy | Precision | Recall | Specificity | F1
Train | 0.9963 | 0.9926 | 0.9985 | 0.9985 | 0.9955
Test | 0.7878 | 0.5779 | 0.6571 | 0.7827 | 0.6100
Hold | 0.8170 | 0.6533 | 0.7329 | 0.8124 | 0.6866
Full | 0.8133 | 0.6409 | 0.7407 | 0.8178 | 0.6808

In most aspects, injection of meta-features yielded better results. On the
hold set, we achieve an average increment of $5.65\%$ over the five metrics.
The least improvement is $3.09\%$ for specificity and the highest is $8.79\%$
for accuracy. This shows that augmenting the images with additional contextual
features helps the model better estimate how each building was impacted by the
disaster. This improvement was achieved despite $32.18\%$ of the data being
missing on average across the meta-features. We believe that domain knowledge
injection can add even more value with better conditioned data and with the
possibility of embedding meta-features at the edge level and not just the node
level.

##  5 Conclusion

Disaster informatics is essential for planning effective responses. With the
availability of satellite images and remote sensing, artificial intelligence
can be valuable for urban data science and disaster planning.

Given that the majority of the destruction is usually clustered around the
epicenter of the disaster, it can be expected that neighboring buildings would
have a similar damage signature. However, in many cases the urban layout of
cities is non-uniform and it is very common to observe neighboring buildings
of different age, height and function. This adds an additional layer of
interactions that makes these buildings sustain damage differently.

In this work, we have presented a graph formulation that allows representing
damage data while connecting buildings. We had proposed BLDNet, a semi-
supervised GCN with a Siamese CNN backbone for extracting local features and
aggregating them with neighboring features. Through semi-supervised learning,
few labeled images enabled BLDNet to obtain predictions in an efficient
manner. We demonstrated the effectiveness of BLDNet on the xBD dataset and
compared it to the multiresolution autoencoder. We also showcased BLDNet’s
effectiveness on the 2020 Beirut Port explosion where augmenting the damage
images with contextual building meta-features improved model performance.

Based on the insights gathered throughout this work, potential directions for
future research include using additional satellite imagery bands in addition
to color bands and applying our graph formulation on geotagged street view
images of damaged buildings.

## Acknowledgments

The authors would like to thank Maxar Technologies for donating satellite
images of the Beirut area during dates surrounding the August 4 explosion. We
also credit the Open Map Lebanon team for promptly providing the building
damage annotations for the Beirut Port explosion.

## References

  * [1] Salman H. Khan, Xuming He, Fatih Porikli, and Mohammed Bennamoun.  Forest Change Detection in Incomplete Satellite Images With Deep Neural Networks.  IEEE Transactions on Geoscience and Remote Sensing, 55(9):5407–5423, September 2017.
  * [2] Bin Tan, Jeffrey G. Masek, Robert Wolfe, Feng Gao, Chengquan Huang, Eric F. Vermote, Joseph O. Sexton, and Greg Ederer.  Improved forest change detection with terrain illumination corrected Landsat images.  Remote Sensing of Environment, 136:469–483, September 2013.
  * [3] Curtis E Woodcock, Scott A Macomber, Mary Pax-Lenney, and Warren B Cohen.  Monitoring large areas for forest change using Landsat: Generalization across space, time and Landsat sensors.  Remote Sensing of Environment, 78(1-2):194–203, October 2001.
  * [4] Komeil Rokni, Anuar Ahmad, Karim Solaimani, and Sharifeh Hazini.  A new approach for surface water change detection: Integration of pixel level image fusion and image classification techniques.  International Journal of Applied Earth Observation and Geoinformation, 34:226–234, February 2015.
  * [5] Ahram Song, Yeji Kim, and Yongil Kim.  Change Detection of Surface Water in Remote Sensing Images Based on Fully Convolutional Network.  Journal of Coastal Research, 91(sp1):426, August 2019.
  * [6] Feng Gao, Xiao Wang, Yunhao Gao, Junyu Dong, and Shengke Wang.  Sea Ice Change Detection in SAR Images Based on Convolutional-Wavelet Neural Networks.  IEEE Geoscience and Remote Sensing Letters, 16(8):1240–1244, August 2019.
  * [7] Yunhao Gao, Feng Gao, Junyu Dong, and Shengke Wang.  Transferred Deep Learning for Sea Ice Change Detection From Synthetic-Aperture Radar Images.  IEEE Geoscience and Remote Sensing Letters, 16(10):1655–1659, October 2019.
  * [8] Anzi Ding, Qingyong Zhang, Xinmin Zhou, and Bicheng Dai.  Automatic recognition of landslide based on CNN and texture change detection.  In 2016 31st Youth Academic Annual Conference of Chinese Association of Automation (YAC), pages 444–448, Wuhan, Hubei Province, China, November 2016. IEEE.
  * [9] Francesca Bovolo and Lorenzo Bruzzone.  A Split-Based Approach to Unsupervised Change Detection in Large-Size Multitemporal Images: Application to Tsunami-Damage Assessment.  IEEE Transactions on Geoscience and Remote Sensing, 45(6):1658–1670, June 2007.
  * [10] Jérémie Sublime and Ekaterina Kalinicheva.  Automatic Post-Disaster Damage Mapping Using Deep-Learning Techniques for Change Detection: Case Study of the Tohoku Tsunami.  Remote Sensing, 11(9):1123, May 2019.
  * [11] R. H. Fraser, R. Fernandes, and R. Latifovic.  Multi-temporal Mapping of Burned Forest over Canada Using Satellite-based Change Metrics.  Geocarto International, 18(2):37–47, June 2003.
  * [12] Chi Zhang, Shiqing Wei, Shunping Ji, and Meng Lu.  Detecting Large-Scale Urban Land Cover Changes from Very High Resolution Remote Sensing Images Using CNN-Based Classification.  ISPRS International Journal of Geo-Information, 8(4):189, April 2019\.
  * [13] Cong Cao, Suzana Dragićević, and Songnian Li.  Land-Use Change Detection with Convolutional Neural Network Methods.  Environments, 6(2):25, February 2019.
  * [14] Dengsheng Lu.  Detection of urban expansion in an urban-rural landscape with multitemporal QuickBird images.  Journal of Applied Remote Sensing, 4(1):041880, September 2010.
  * [15] Hannah Rae Kerner, Kiri L. Wagstaff, Brian D. Bue, Patrick C. Gray, James F. Bell, and Heni Ben Amor.  Toward Generalized Change Detection on Planetary Surfaces With Convolutional Autoencoders and Transfer Learning.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(10):3900–3918, October 2019.
  * [16] Stefan A. Robila.  Use of Remote Sensing Applications and its Implications to the Society.  In 2006 IEEE International Symposium on Technology and Society, pages 1–6, Queens, NY, USA, June 2006. IEEE.
  * [17] Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston.  xBD: A Dataset for Assessing Building Damage from Satellite Imagery.  arXiv:1911.09296 [cs], November 2019.
  * [18] Anju Asokan and J. Anitha.  Change detection techniques for remote sensing applications: a survey.  Earth Science Informatics, 12(2):143–160, June 2019.
  * [19] Wenzhong Shi, Min Zhang, Rui Zhang, Shanxiong Chen, and Zhao Zhan.  Change Detection Based on Artificial Intelligence: State-of-the-Art and Challenges.  Remote Sensing, 12(10):1688, May 2020.
  * [20] Lazhar Khelifi and Max Mignotte.  Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis.  arXiv:2006.05612 [cs], June 2020.
  * [21] Daifeng Peng, Yongjun Zhang, and Haiyan Guan.  End-to-End Change Detection for High Resolution Satellite Images Using Improved UNet++.  Remote Sensing, 11(11):1382, June 2019.
  * [22] Yang Zhan, Kun Fu, Menglong Yan, Xian Sun, Hongqi Wang, and Xiaosong Qiu.  Change Detection Based on Deep Siamese Convolutional Network for Optical Aerial Images.  IEEE Geoscience and Remote Sensing Letters, 14(10):1845–1849, October 2017.
  * [23] Rodrigo Caye Daudt, Bertr Le Saux, Alexandre Boulch, and Yann Gousseau.  Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks.  In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pages 2115–2118, Valencia, July 2018. IEEE.
  * [24] Haobo Lyu, Hui Lu, and Lichao Mou.  Learning a Transferable Change Rule from a Recurrent Neural Network for Land Cover Change Detection.  Remote Sensing, 8(6):506, June 2016.
  * [25] Lichao Mou, Lorenzo Bruzzone, and Xiao Xiang Zhu.  Learning Spectral-Spatial-Temporal Features via a Recurrent Convolutional Neural Network for Change Detection in Multispectral Imagery.  IEEE Transactions on Geoscience and Remote Sensing, 57(2):924–935, February 2019.
  * [26] Hongruixuan Chen, Chen Wu, Bo Du, Liangpei Zhang, and Le Wang.  Change Detection in Multisource VHR Images via Deep Siamese Convolutional Multiple-Layers Recurrent Neural Network.  IEEE Transactions on Geoscience and Remote Sensing, 58(4):2848–2864, April 2020.
  * [27] Ran Jing, Shuang Liu, Zhaoning Gong, Zhiheng Wang, Hongliang Guan, Atul Gautam, and Wenji Zhao.  Object-based change detection for VHR remote sensing images based on a Trisiamese-LSTM.  International Journal of Remote Sensing, 41(16):6209–6231, August 2020.
  * [28] Shunping Ji, Yanyun Shen, Meng Lu, and Yongjun Zhang.  Building Instance Change Detection from Large-Scale Aerial Images using Convolutional Neural Networks and Simulated Samples.  Remote Sensing, 11(11):1343, June 2019.
  * [29] Francesco Nex, Diogo Duarte, Fabio Giulio Tonolo, and Norman Kerle.  Structural Building Damage Detection with Deep Learning: Assessment of a State-of-the-Art CNN in Operational Conditions.  Remote Sensing, 11(23):2765, November 2019.
  * [30] Huiwei Jiang, Xiangyun Hu, Kun Li, Jinming Zhang, Jinqi Gong, and Mi Zhang.  PGA-SiamNet: Pyramid Feature-Based Attention-Guided Siamese Network for Remote Sensing Orthoimagery Building Change Detection.  Remote Sensing, 12(3):484, February 2020.
  * [31] Bahareh Kalantar, Naonori Ueda, Husam A. H. Al-Najjar, and Alfian Abdul Halin.  Assessment of Convolutional Neural Network Architectures for Earthquake-Induced Building Damage Detection based on Pre- and Post-Event Orthophoto Images.  Remote Sensing, 12(21):3529, October 2020.
  * [32] Hiroyuki Miura, Tomohiro Aridome, and Masashi Matsuoka.  Deep Learning-Based Identification of Collapsed, Non-Collapsed and Blue Tarp-Covered Buildings from Post-Disaster Aerial Images.  Remote Sensing, 12(12):1924, June 2020.
  * [33] Bradley J. Wheeler and Hassan A. Karimi.  Deep Learning-Enabled Semantic Inference of Individual Building Damage Magnitude from Satellite Images.  Algorithms, 13(8):195, August 2020.
  * [34] Thomas N. Kipf and Max Welling.  Semi-Supervised Classification with Graph Convolutional Networks.  arXiv:1609.02907 [cs, stat], February 2017.
  * [35] Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot.  Graph Convolutional Networks for Hyperspectral Image Classification.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–13, 2020.
  * [36] Ushasi Chaudhuri, Biplab Banerjee, and Avik Bhattacharya.  Siamese graph convolutional network for content based remote sensing image retrieval.  Computer Vision and Image Understanding, 184:22–30, July 2019.
  * [37] Sudipan Saha, Francesca Bovolo, and Lorenzo Bruzzone.  Semisupervised Change Detection Using Graph Convolutional Network.  IEEE Geoscience and Remote Sensing Letters, 18(4):607 – 611, 2020\.
  * [38] Qing Wang, Xiaodong Zhang, Guanzhou Chen, Fan Dai, Yuanfu Gong, and Kun Zhu.  Change detection based on Faster R-CNN for high-resolution remote sensing images.  Remote Sensing Letters, 9(10):923–932, October 2018.
  * [39] Ethan Weber and Hassan Kané.  Building Disaster Damage Assessment in Satellite Imagery with Multi-Temporal Fusion.  arXiv:2004.05525 [cs], April 2020.
  * [40] Jinhua Su, Yanbing Bai, Xingrui Wang, Dong Lu, Bo Zhao, Hanfang Yang, Erick Mas, and Shunichi Koshimura.  Technical Solution Discussion for Key Challenges of Operational Convolutional Neural Network-Based Building-Damage Assessment from Satellite Imagery: Perspective from Benchmark xBD Dataset.  Remote Sensing, 12(22):3808, November 2020.
  * [41] Joseph Z. Xu, Wenhan Lu, Zebo Li, Pranav Khaitan, and Valeriya Zaytseva.  Building Damage Detection in Satellite Imagery Using Convolutional Neural Networks.  arXiv:1910.06444 [cs, eess, stat], October 2019.
  * [42] Vitus Benson and Alexander Ecker.  Assessing out-of-domain generalization for robust building damage detection.  arXiv:2011.10328 [cs], November 2020.
  * [43] Yanbing Bai, Junjie Hu, Jinhua Su, Xing Liu, Haoyu Liu, Xianwen He, Shengwang Meng, Erick Mas, and Shunichi Koshimura.  Pyramid Pooling Module-Based Semi-Siamese Network: A Benchmark Model for Assessing Building Damage from xBD Satellite Imagery Datasets.  Remote Sensing, 12(24):4055, December 2020.
  * [44] Wanting Yang, Xianfeng Zhang, and Peng Luo.  Transferability of Convolutional Neural Network Models for Identifying Damaged Buildings Due to Earthquake.  Remote Sensing, 13(3):504, January 2021.
  * [45] Zhuo Zheng, Yanfei Zhong, Junjue Wang, Ailong Ma, and Liangpei Zhang.  Building damage assessment for rapid disaster response with a deep object-based semantic change detection framework: From natural disasters to man-made disasters.  Remote Sensing of Environment, 265:112636, November 2021.
  * [46] Chinmayee Pati, Ashok K. Panda, Ajaya Kumar Tripathy, Sateesh K. Pradhan, and Srikanta Patnaik.  A novel hybrid machine learning approach for change detection in remote sensing images.  Engineering Science and Technology, an International Journal, 23(5):973–981, October 2020.
  * [47] Sofia Tilon, Francesco Nex, Norman Kerle, and George Vosselman.  Post-Disaster Building Damage Detection from Earth Observation Imagery Using Unsupervised and Transferable Anomaly Detecting Generative Adversarial Networks.  Remote Sensing, 12(24):4193, December 2020.
  * [48] Daifeng Peng, Lorenzo Bruzzone, Yongjun Zhang, Haiyan Guan, Haiyong Ding, and Xu Huang.  SemiCDNet: A Semisupervised Convolutional Neural Network for Change Detection in High Resolution Remote-Sensing Images.  IEEE Transactions on Geoscience and Remote Sensing, 59(7):5891–5906, July 2021.
  * [49] Nagma Khan, Ushasi Chaudhuri, Biplab Banerjee, and Subhasis Chaudhuri.  Graph convolutional network for multi-label VHR remote sensing scene recognition.  Neurocomputing, 357:36–46, September 2019.
  * [50] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-Label Image Recognition With Graph Convolutional Networks.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5172–5181, Long Beach, CA, USA, June 2019. IEEE.
  * [51] Lichao Mou, Xiaoqiang Lu, Xuelong Li, and Xiao Xiang Zhu.  Nonlocal Graph Convolutional Networks for Hyperspectral Image Classification.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–12, 2020.
  * [52] D. T. Lee and B. J. Schachter.  Two algorithms for constructing a Delaunay triangulation.  International Journal of Computer & Information Sciences, 9(3):219–242, June 1980.
  * [53] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [54] Marco Calderisi, Gabriele Galatolo, Ilaria Ceppa, Tommaso Motta, and Francesco Vergentini.  Improve Image Classification Tasks Using Simple Convolutional Architectures with Processed Metadata Injection.  In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pages 223–230, Sardinia, Italy, June 2019. IEEE.
  * [55] Jeffrey S. Ellen, Casey A. Graff, and Mark D. Ohman.  Improving plankton image classification using context metadata.  Limnology and Oceanography: Methods, 17(8):439–461, August 2019\.
  * [56] Alice Bucknell.  High contrast: We chart beirut’s ever-changing architecture scene, Aug 2018.
  * [57] Lebanon: Beirut port explosions - aug 2020.
  * [58] S. E. Rigby, T. J. Lodge, S. Alotaibi, A. D. Barr, S. D. Clarke, G. S. Langdon, and A. Tyas.  Preliminary yield estimation of the 2020 Beirut explosion using video footage from social media.  Shock Waves, September 2020.
  * [59] Alaa Krayem, Aram Yeretzian, Ghaleb Faour, and Sara Najem.  Machine learning for buildings’ characterization and power-law recovery of urban metrics.  PLOS ONE, 16(1):e0246096, January 2021.
  * [60] Matthias Fey and Jan Eric Lenssen.  Fast Graph Representation Learning with PyTorch Geometric.  arXiv:1903.02428 [cs, stat], April 2019.
  * [61] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * [62] Dino Ienco and Ruggero G. Pensa.  Semi-Supervised Clustering With Multiresolution Autoencoders.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1–8, Rio de Janeiro, July 2018. IEEE.
  * [63] C. E. Shannon.  A mathematical theory of communication.  The Bell System Technical Journal, 27(3):379–423, 1948.
  * [64] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [65] Student.  The probable error of a mean.  Biometrika, pages 1–25, 1908.
  * [66] Frank Wilcoxon.  Individual comparisons by ranking methods.  Biometrics Bulletin, 1(6):80–83, 1945.