[1,3]Necip Enes Gengec

[1]Graduate School of Engineering and Technology, Istanbul Technical
University, Istanbul, Turkey

2]Department of Geomatics Engineering, Istanbul Technical University,
Istanbul, Turkey

3]Machine and Hybrid Intelligence Lab, Northwestern University, Chicago,
Illinois, USA

# AI Powered Road Network Prediction with Multi-Modal Data

<EMAIL_ADDRESS>Ergin Tari<EMAIL_ADDRESS>Ulas Bagci
<EMAIL_ADDRESS>*  [  [

###### Abstract

This study presents an innovative approach for automatic road detection with
deep learning, by employing fusion strategies for utilizing both lower-
resolution satellite imagery and GPS trajectory data, a concept never explored
before. We rigorously investigate both early and late fusion strategies, and
assess deep learning based road detection performance using different fusion
settings. Our extensive ablation studies assess the efficacy of our framework
under diverse model architectures, loss functions, and geographic domains
(Istanbul and Montreal). For an unbiased and complete evaluation of road
detection results, we use both region-based and boundary-based evaluation
metrics for road segmentation. The outcomes reveal that the ResUnet model
outperforms U-Net and D-Linknet in road extraction tasks, achieving superior
results over the benchmark study using low-resolution Sentinel-2 data. This
research not only contributes to the field of automatic road detection but
also offers novel insights into the utilization of data fusion methods in
diverse applications.

###### keywords:

Road detection, GPS Trajectory, Multi-modal data, Data Fusion, Deep Learning

##  1 Introduction

Digital maps are used in wide range of applications including navigation,
urban planning, disaster management and response, and many more with ”road
network data” serving as a primary component of these maps [1, 2, 3]. Road
network data can be produced manually through digitization or field surveys,
crowd-sourced, or automatically detected through aerial/satellite imagery
and/or using GPS trajectories.

While its significance is bold, analyzing road network data can be quite
challenging with manual efforts. Hence, automatic detection of road networks
from images has recently been adopted due to its cost efficiency. The success
of emerging artificial intelligence (AI)/deep learning (DL) methods has played
a primary role in this switch [4]. In these applications, the first and the
major step is to segment (delineate) satellite or aerial images using
supervised deep learning models. High-resolution satellite imagery is more
often used and desired in such applications than other imaging modalities [5,
3, 6, 7] but automated methods with high-resolution satellite imagery is still
costly. Because of high cost of such images, using lower-resolution imagery
such as freely available Sentinel-2 [8, 9] becomes an attractive research area
with a few existing studies. The use of low-resolution images presents certain
challenges including having lower resolution for details, thus low accuracy in
quantitative measures coupled to it. However, it provides also potential
opportunities to research on. For example, Sentinel-2 is freely available and
can provide broad coverage (more global). Further, Sentinel-2 provides a
better temporal resolution and it is multi-spectral in nature (i.e., capturing
several spectral bands). Practical and cost-effective nature of the low-
resolution imagery is opening new and unexplored doors for research community.
That being said, current efforts in this domain and particularly in road
detection tasks is limited and in early steps; further research on improving
the automatic road detection task with lower-resolution data can provide more
cost-effective solutions. In this paper, our effort is within this research
line: we aim to develop cost-effective AI solution for road network prediction
with multi-modal data.

”GPS trajectory data” is another source used in road network segmentation.
Different methods are used to detect roads, including point clustering [10],
kernel density estimation (KDE) [11], graph-based road generation [12], and
point matching [13]. In addition, deep learning methods are used for road
segmentation over rasterized GPS trajectory data fusion with satellite imagery
[14]. High-resolution satellite imagery is still an expensive choice in these
cases. To our knowledge, no study has been conducted yet using lower-
resolution satellite imagery and GPS trajectory fusion for automatic road
detection. In different fields, fusion operations are commonly used in ad-hoc
manner. For example, early fusion (Figure 1b) is the prominent method in
combining varying data sources. However, optimal fusion strategy is often
unknown especially when data sources have some common overlaps. In other
words, the effect of fusion at later stages and success of alternative fusion
operations on segmentation is largely unknown. We speculate that exploring
such gaps may improve the road detection task.

The overall goal of this study is therefore to introduce an innovative
approach for automatic road detection and segmentation by fusing lower-
resolution satellite imagery with GPS trajectory data, an area yet unexplored
in the current landscape of studies. We will investigate both early and late
fusion strategies for low resolution satellite imagery with GPS trajectory
data (Figure 1c and 1d) and explore road segmentation performance in depth
using relatively lower-resolution satellite imagery in different fusion
settings. In our ablation studies, the efficacy of this framework is tested
under various settings of model architectures and loss functions in different
geographic domains.

Figure 1: (a) Baseline model with satellite image input, (b) baseline model
with early fusion of satellite image and GPS trajectory input, (c) late fusion
Type-1 (model applied only to satellite image stream) with satellite image and
GPS trajectory input, (d) late fusion Type-2 (model applied to both satellite
image and GPS trajectory streams) with satellite image and GPS trajectory
input.

##  2 Related Work

The automatic detection of road networks has become an increasingly popular
research topic due to its practical applications [4, 1]. In recent years, many
studies have focused on the use of deep learning methods to extract road
networks from various data sources such as satellite imagery and GPS
trajectory data [14, 15, 16, 17]. Even with deep learning, advanced artificial
intelligence methodologies solving complex problems at scale with highly
accurate manner, the problem is still solved at sub-optimally pace because of
highly variable image qualities across different data sources, even within the
data source, complexity of the road features, problems such as occlusion,
lighting differences, and other similar-looking features. To this end,
existing studies presented several fusion methods to integrate GPS trajectory
data into satellite imagery to improve the accuracy of road extraction. Some
studies focused on exploring different loss functions that might be more
suitable for the road extraction tasks. Last, but not least, it is worth to
revisit the evaluation metrics for deep learning based segmentation strategies
as they are crucial for measuring the effectiveness of different road
extraction methods. In this section, we provide a review of the relevant
literature in all these areas.

###  2.1 Road extraction using satellite imagery with deep learning

Mnih and Hinton’s pioneering work (2010) was the first significant study to
apply deep neural networks to road extraction from satellite imagery. Since
then, many studies have used different deep network architectures to improve
the performance of road extraction. U-Net, the widely used U-shaped deep
network, was originally developed for medical image segmentation [18]. Later,
it has been applied to road segmentation in different studies. Literature
becomes increasingly vast in methods that relying on U-Net. For instance,
Residual U-Net (ResUnet) has been used in road segmentation with satellite
imagery which is one variant of U-Net that uses residual units to enhance
segmentation results [6]. D-Linknet is another U-shaped network that uses
dilated convolutions and has been frequently used as a benchmark model [2].
Other notable studies are BiHRnet [19], HsgNet [20], RADANet [21], SDUNet
[22], and the study of [23]. In more recent years, we are witnessing a huge
swamp from CNN based architectures to Transformers based architectures due to
their self-attention mechanisms and better performances when the architectures
are not-so-deep. Despite their success, there is a high computational burden
in Transformers as well as more data requirement, not allowing them to be
easily adapted for multidimensional data. BDTNet [24], RoadFormer [25], and
Seg-Road [26] are recent examples of the latest transformer-based models that
have been applied to road segmentation.

In the context of image resolution, deep learning-based approaches are already
applied to high-resolution satellite imagery [5, 6, 19, 20, 21, 22, 23, 24,
25]. There are only a few approaches that use lower resolution satellite
imagery such as [8], [27] and [9]. These examples use Sentinel-2 data for road
extraction as an input to either U-Net or HRNet.

###  2.2 Road extraction using satellite imagery and GPS Trajectory with deep
learning

Deep learning architectural engineering becomes a de facto strategy for
improved road segmentation performance [4]. For instance, [14] proposed a
U-shaped architecture with 1D convolution, where satellite imagery and GPS
trajectory data are fed into the network as concatenated image layers. [28]
used a similar approach, utilizing a U-Net model with refined labels.
D-Linknet, which incorporates concatenated satellite imagery and GPS
trajectory data, has been frequently used and extended in recent studies such
as FuNet [29], RING-Net [30], and [31]. Other studies have proposed novel
techniques to incorporate GPS trajectory data, such as [15], [16], and [17].
These studies demonstrate the potential benefits of combining GPS trajectory
data with satellite imagery for improved road segmentation accuracy. Despite
their benefits, none of these studies have reported using lower resolution
satellite imagery in conjunction with GPS trajectory data. Also, 1D
convolution is more appropriate for GPS trajectory data while not for imaging
data indicating potential sub-optimality in fusing the data.

###  2.3 Multi modal data fusion

Multi-modal data can be fused within deep learning models. Theoretically, the
fusion process can occur at various stages within the model, employing
different fusion methods. These fusion stages can be categorized as early,
late and hybrid fusion [32]. In early fusion, the fusion takes place at the
beginning of the model (Figure 1b) where as late fusion occurs at the end,
just before the output layer (Figure 1c and 1d). Hybrid fusion involves a more
complex flow and can be summarized as fusion that takes place at the
intermediate stages of the model. When considering fusion methods, multiple
matrix operations can be utilized based on the desired outcome, often
involving a trial and error. Concatenation is the most frequently preferred
fusion method [14, 15, 16, 17].

In the context of road extraction using satellite imagery and GPS trajectory,
different stages of fusion methods have been tested. [15] proposed their own
method and evaluated its accuracy in comparison to early and late fusion
alternatives, utilizing concatenation as the fusion method. Their study found
that early fusion provided slightly better IoU results when compared to late
fusion. In another study, [16] examined early and late fusion in their
DeepDualMapper study. They employed concatenation for early fusion and
averaging for late fusion as the fusion method. Similar to [15], [16] achieved
the superior results with early fusion. Furthermore, [17] explored early,
deep, and vanilla fusion in their study. Deep fusion represents an example of
hybrid fusion while vanilla fusion is a late fusion variant that employs
intersection as the fusion method. In this study, early fusion outperformed
vanilla fusion in terms of recall and $F_{1}$. Literature shows that research
on fusion stages is limited. As a note, the fusion methods utilized in these
studies mostly revolve around concatenation only.

###  2.4 Loss functions

Loss functions are needed in the optimization of deep neural networks [33].
Numerous loss functions have been proposed according to the specific task at
hand. In the context of road extraction, mean square error (MSE) [6] and
binary cross-entropy (BCE) [1] are two commonly used functions. BCE is
generally regularized with an additional loss function such as Dice [7, 2, 20]
or $L_{2}$ norm [19]. [3] employed a focal loss function, a BCE variant that
addresses class imbalance issues. Furthermore, researchers have proposed
application-specific tailored loss functions by combining multiple loss
functions [17, 30, 28] when necessary. To our best of knowledge, no study has
been conducted to comprehensively evaluate their performance in road
extraction using deep learning. Our study fills this research gap.

###  2.5 Evaluation metrics

Evaluation metrics are essential for monitoring the performance of a given
model. Various metrics have been adopted in segmentation tasks [34, 35] in
general. Precision and recall are considered as fundamental metrics in many
road extraction studies. These metrics are often employed alongside additional
metrics such as the $F_{1}$ score and/or intersection over union (IoU) [17,
20, 19] in practice. Precision and recall are used to calculate the $F_{1}$
score, which is calculated by the harmonic mean of these two metrics. IoU
represents the ratio between the intersection and the union of the ground
truth and predicted segments. In some studies, IoU is used as the sole metric
[2, 14, 7]. Occasionally, custom metrics are employed, such as the break-even
point/relaxed precision [6], global IoU [15] or average path length similarity
(APLS) [36, 37, 19]. However, the adoption of these metrics remains limited.

[38] developed a framework to guide the selection of appropriate metrics for
different machine learning tasks. For segmentation tasks, the framework
suggests using an region-based metric such as IoU or $F_{1}$ score,
complemented by a boundary-based metric. The inclusion of a boundary-based
metric helps to address the issues caused by the lack of shape awareness in
region-based metrics. Notably, there are no literature examples of boundary-
based metrics being used in the context of road extraction. In order to
complement full evaluation spectrum, this metric and region-based metric are
comprehended in our study.

###  2.6 Benchmark dataset

Benchmarking serves the purpose of facilitating fair comparisons and
validations among different models under the same conditions, thereby enabling
the identification of strengths and weaknesses. Several benchmark dataset are
available for road extraction from satellite imagery [4]. Massachusetts [5],
DeepGlobe [3] and SpaceNet [37] dataset are the leading examples which are
widely used as benchmark. These dataset comprise high-resolution satellite
imagery. In couple couple of research satellite imagery extracted from Google
Maps API from different zoom levels is used in road extraction. [39] used the
Google Maps API to obtain satellite imagery for the road extraction task in
Istanbul, while [16] acquired data from Porto, Shanghai and Singapore in the
same method and they conducted their study with additional GPS trajectory
data. [8, 27] conducted road extraction study using low-resolution Sentinel-2
data.

The benchmark dataset available in the literature are predominantly based on
high-resolution imagery, which can be costly to acquire for real-world
applications. Approaches such as those employed by [39] and [16] are not
viable for all scenarios. The utilization of freely available low-resolution
Sentinel-2 data or similar low-resolution satellite imagery sources is
noteworthy, although the availability of GPS trajectory data is essential to
support research in the multi-modal domain. Furthermore, such dataset should
cover multiple geographies to enhance studies that measure the
generalizability of model performance across different dataset. Our study
provides a benchmark dataset which consist of low-resolution satellite imagery
and GPS trajectory data from two different locations which is filling the gap
in literature.

###  2.7 Our contributions

The main novelty of our study lies in its innovative use of lower-resolution
satellite imagery and GPS trajectory fusion for road detection and
quantification via segmentation. In the light of relevant studies and their
limitations, our study has the following major contributions:

  1. 1.

We extensively investigate the impact of GPS trajectory data on road
extraction using low-resolution satellite imagery (Sentinel-2). Through this,
we anticipate to initiate a new wave of studies focused on exploiting lower-
resolution image and GPS trajectory data, ultimately contributing broader
advance of automatic road detection methods.

  2. 2.

We carefully design fusion architectures (early fusion, late fusion Type-1/2)
consisting of the state-of-the-art architectures (U-Net, ResUnet, D-Linknet)
with various loss functions (MSE, BCE, Focal loss) using both Sentinel-2 data
and GPS trajectory data. The fusion architectures is expected to amplify the
efficacy of road detection.

  3. 3.

We assess the fusion performance of the ablation models by employing fusion
techniques at different stages and utilizing various fusion methods (e.g.,
early fusion, late fusion) as illustrated in Figure 1.

  4. 4.

We provide a novel benchmark dataset and test the generalization ability of
the models on a newly developed benchmark dataset that incorporates multi-
modal data, including GPS trajectory and Sentinel-2 data from diverse
geographic locations (Istanbul and Montreal).

  5. 5.

Due to inherent limitations of traditional evaluation metrics for segmentation
tasks, we postulate a full spectrum segmentation evaluation strategy by using
both region and boundary-based metrics, giving broader understanding of
segmentation methods under various conditions. We propose to use both region
(IoU), and boundary based methods (Boundary-IoU) together to give a better
understanding and fair evaluation of methods.

By addressing these objectives, our study aims to (1) explore the influence of
GPS trajectory data by (2) evaluating different deep learning architectures,
(3) comparing loss functions, (4) analyzing fusion techniques, and their
generalization capabilities, and (5) applying a new type of evaluation metric
in the road extraction research.

##  3 Methodology

In this section, we delve into the methodology employed to accomplish the
research objectives of this study.

###  3.1 Choosing segmentation models

Based on the state of the art algorithms, we employed U-Net, ResUnet and
D-Linknet in order to assess their strengths and weaknesses in the road
extraction using satellite imagery and GPS trajectory data. Briefly, these
methods are described as follows.

U-Net is a convolutional neural network architecture that incorporates both
convolutional and up-convolutional layers, connected by skip connections [18].
It consists of an encoder, a bottleneck, a decoder, and skip connections
between the encoder and decoder parts. Although initially developed for
biomedical image segmentation, U-Net has been successfully applied in various
domains. The original U-Net is trained on the RGB data, where each color layer
is stacked into a 3D tensor, yielding binary predictions.

ResUnet is a variant of U-Net specifically designed for road extraction from
satellite imagery. ResUnet improves upon U-Net by incorporating residual units
[6]. Residual learning or residual unit, first introduced by [40], addresses
the problem of overfitting in large deep neural networks. In ResUnet, the
plain neural units in U-Net are replaced with identity-mapped replicas of the
same units, known as residual units [6]. This addition leads to significant
improvement in IoU.

D-Linknet is another U-shaped segmentation model developed for road
extraction, building upon the success of its predecessor, Linknet [2].
D-Linknet introduces a dilated convolution convolution block in the bottleneck
of U-Net along with the residual units. Additionally, D-Linknet leverages
transfer learning, where the encoder part of the model is initialized with a
ResNet34 pretrained on the ImageNet dataset. D-Linknet achieved the best
results in the DeepGlobe Road Extraction Challenge - 2018 [3], and subsequent
improvements have been made by other researchers [20, 19].

###  3.2 Loss functions details

Mean square error (MSE), binary cross-entropy (BCE), and focal loss were
utilized in this study to train the networks and assess their performance in
road extraction tasks.

MSE, which is an example of mean bias error (MBE) losses [41], is calculated
as the sum of squared errors between predictions and the ground truth. It can
be defined by the following equation [6]:

$L_{MSE}(W)=\frac{1}{N}\sum_{i=1}^{N}\|Net(I_{i};W)-s_{i}\|^{2},$ (1)

where $Net(I_{i};W)$ represents the segmentation, $s_{i}$ denotes the ground
truth, and $N$ is the number of training examples.

BCE is a probabilistic loss function [41] used to measure the difference
between two probability distributions [42]. It is defined as:

$L_{BCE}(y,\hat{y})=-(y\log{(\hat{y})}+(1-y)\log{(1-\hat{y})}),$ (2)

where $y$ represents the ground truth and $\hat{y}$ represents the
predictions.

In the context of segmentation tasks, the available classes in the data are
often imbalanced. For example, in the road extraction, foreground pixels are
more frequent when compared to road pixels. Focal loss is a loss function
designed to address such class imbalance [43]:

$L_{FL}(p_{t})=-(1-p_{t})^{\gamma}\log(p_{t}).$ (3)

In the focal loss equation, $log(p_{t})$ represents cross-entropy,
$(1-p_{t})^{\gamma}$ denotes the modulation factor and $\gamma$ is the
focusing factor. The optimized parameters for focal loss are $\gamma=2$ and
$(1-p_{t})=0.25$ [43].

###  3.3 Structuring multi-modal data fusion

The different stages of fusion are demonstrated in Figure 1. In the examples
without fusion the satellite imagery is directly fed into the model (Figure
1a). In early fusion, both the satellite imagery and GPS trajectory data are
fused and then fed into the model (Figure 1b). In the late fusion, both
dataset are fed into in two separate models:

  * •

Type - 1: Deep learning model applied to satellite imagery but not applied to
GPS trajectory (Figure 1c).

  * •

Type - 2: Deep learning model applied to both the satellite imagery and GPS
trajectory (Figure 1d).

After the late fusion networks, the two streams of data are combined into one
using a fusion operation. Fusion methods involve matrix operations that
combine multiple data sources into one. Table 1 summarizes the fusion
operations that are used in this study along with their respective equations.

Table 1: Fusion operators: A and B are input and C is the resulting tensor. Fusion Operator | Equation | Fusion Stage
---|---|---
Concatenate | $C=[A\|B]$ | Early/Late
Average | $C=(A+B)\;\circ\;0.5$ | Late
Maximum | $C=max(A,B)$ | Late
Multiply | $C=A\;\circ\;B$ | Late
|  |

###  3.4 Evaluation metric details

As recommended for segmentation tasks by [38], the region-based metric IoU and
the boundary-based metric Boundary-IoU [44] are adopted as the evaluation
metric in this study.

The IoU of an individual example ($i$) is defined by the following equation
[34]:

$IoU_{i}=\frac{True\>Positives_{i}}{True\>Positives_{i}+False\>Positives_{i}+False\>Negatives_{i}}.$
(4)

All values of this equation are in the number of pixels.

Boundary-IoU is a special form of the IoU metric. To calculate Boundary-IoU,
the boundary pixels of the class are first extracted, and then the IoU metric
is calculated using the same equation. Boundary IoU defined with the following
equation:

$Boundary\>IoU_{i}=\frac{(G_{d}\cap G)\cap(P_{d}\cap P)}{(G_{d}\cap
G)\cup(P_{d}\cap P)},$ (5)

where $G$ represents ground truth, $P$ represents prediction and $d$
represents the contour distance from mask pixels [44].

The performance evaluation of the models are conducted using multiple test
images. The mean value of IoU (mIoU) and Boundary-IoU (mBoundary-IoU) are
considered as the final metric values and are calculated using the following
equation [20]:

$mIoU=\frac{1}{n}\sum_{n}^{i=1}{IoU_{i}}.$ (6)

##  4 Experiments

Experiments carried out in two different area for this study. This section
provides the details about data, pre-processing steps, details of
implementation of the methods explained in Section 3, the summary of the
results and additional analysis.

###  4.1 Data and pre-processing

Experiments were conducted in Istanbul - Turkey and Montreal - Canada. The
work areas and corresponding road network coverage can be seen in Figure 2.
These areas were chosen due to the availability of both GPS trajectory data
and the Sentinel-2 data.

(a) Input data: Istanbul test set

(b) Input data: Montreal test set

Figure 2: Istanbul and Montreal work area.

The GPS trajectory data in Istanbul was obtained from [45]. The data contains
approximately 360 million GPS points from different months of 2020 and is
collected from various types of vehicles such as cars and trucks. The data for
Montreal was shared by [46] and contains data from 2016 and 2017. The data
consists of 40 million GPS points derived from passenger cars.

The satellite imagery used in experiments is derived from Sentinel-2 [47].
Sentinel-2 provides low-resolution ($10m/pixel$) multi-spectral satellite
imagery, including red, green, blue (RGB) and infrared bands. The
corresponding Sentinel-2 images taken around the same period as the GPS
trajectory data were used in this study.

Since this study involves a supervised learning, a labeled data is required.
Open Street Map (OSM) is an open map data source which is developed and
maintained by volunteers [48]. The OSM data has been used in various road
extraction studies [49, 27, 4]. In this study, the label data was created
using OSM data.

Figure 3: Data pre-processing details.

The data underwent pre-processing steps before training of the deep learning
models. The details of the pre-processing steps are summarized in Figure 3.
The RGB and infrared bands (RGB-I) were extracted from Sentinel-2 data and
upscaled to $2.5m$ resolution using the cubic convolution re-sampling method,
similar to [27]. After upscaling, all bands were normalized to range of 0-1.

The GPS trajectory data was stored in tabular form and needed to be rasterized
[50]. To maintain the same resolution as the Sentinel-2 data, the GPS
trajectory data was rasterized into $2.5m$ resolution imagery. The resultant
imagery contains the frequency of GPS points per $2.5m$ x $2.5m$ square
pixels.

The OSM data was stored in vector data format. To use it in this study, the
OSM data was also rasterized. Since different classes of roads have different
widths, a varying buffer was applied to the vector data, and rasterization was
applied to the buffered data. The buffer values per road class are summarized
in Table 2.

Table 2: Road classes and applied buffer size. Buffer (m) | OSM Road Class (fclass)
---|---
10 | ”motorway”, ”primary”, ”secondary”
6 | Remaining classes
4 |  ”footway” , ”track”, ”service” , ”steps” , ”track_grade1” , ”track_grade2” , ”track_grade3” , ”track_grade4” , ”track_grade5” , ”track”, ”bridleway”
|

All preprocessed data used in this study have been made available online to
enable reproducibility and to be used as a benchmark in similar studies111The
pre-processed data can be downloaded from following URL:
https://github.com/nagellette/sentinel_traj_nn/blob/master/Data.md

###  4.2 Implementation and experimentation details

This section provides information about the implementation of methods and
additional details regarding the training of deep learning models that are
used in the experiments.

The U-Net [18], ResUnet [6] and D-Linknet [2] models were implemented from
scratch using the TensorFlow framework [51], and their respective architecture
details were adopted from the corresponding publications. Additionally, the
proposed to fusion stages were implemented using the same model architecture
after adoption to the corresponding fusion model flow. The loss functions, MSE
and BCE, were used as provided in TensorFlow framework. For focal loss, the
implementation from TensorFlow Addons [52] was used with the default
parameters specified in Section 3.2. The IoU metric was utilized as
implemented in TensorFlow, while the Boundary-IoU metric [44] was implemented
from scratch as it is explained in Section 3.4.

Both dataset were split into patches of size $512$ x $512$ pixels. To increase
the dataset size, the raw patches were rotated at the angels of 45, 90, 135,
180, 225, 270 and 315 degrees with a 20% overlap between neighboring patches.
The total patch count for Istanbul and Montreal reached to 20,000 patches. The
data was divided into train, validation, and test sets, with a ratio of 60%,
20% and 20% ratio respectively.

All experiments were conducted using an NVIDIA Tesla V100 GPU with 16GB RAM.
The models were trained with the Adam optimizer as it is implemented in
TensorFlow, with a learning rate set to $0.001$. Training was performed using
batches of patches which were randomly selected from the training set. The
number of training epochs and batch sizes varied depending on the convergence
of different models at different epochs and due to the memory limitations
caused by the large model size for ResUnet and D-Linknet in late fusion
experiments. Table 3 summarizes the batch size, the number of epochs, and the
number of batches per epoch used in the experiments. The training procedure
was validated at the end of each epoch using $200$ randomly selected batches
from the validation set. Once the training was completed, the performance of
each model was evaluated using the IoU and Boundary-IoU metrics on $1000$
samples from the test set.

Cross work area training and testing were conducted to assess the
generalization performance of the models on different dataset. For this
purpose, the same model was trained using separately for Istanbul and
Montreal, and Istanbul+Montreal together, and each of these trained models
were evaluated on three test data combinations. For example, if a model was
trained with data from Istanbul, it was evaluated using Istanbul, Montreal and
Istanbul+Montreal test data where Istanbul+Montreal contains 50% data from
Istanbul test set and 50% from Montreal test set.

Table 3: The model training details: batch, epoch and number of batches per epochs in different experiments. Model | Fusion Stage | Batch Size | Epochs | # of batches/epoch
---|---|---|---|---
U-Net | Early | 4 | 80 | 500
ResUnet | Early | 4 | 80 | 500
D-Linknet | Early | 4 | 150 | 500
U-Net | Late, Type-1 | 4 | 80 | 500
ResUnet | Late, Type-1 | 2 | 80 | 1000
D-Linknet | Late, Type-1 | 2 | 150 | 1000
U-Net | Late, Type-2 | 4 | 80 | 500
ResUnet | Late, Type-2 | 2 | 80 | 1000
D-Linknet | Late, Type-2 | 2 | 150 | 1000
|  |  |  |

All implementations used in this study have been made available online to
enable reproducibility222The implementations of the methods and experiments
can be downloaded from the following URL:
https://github.com/nagellette/sentinel_traj_nn.

###  4.3 Results

In the experiments, we considered Sentinel-2 only and early fusion as baseline
results. The experiment results are summarized in the following tables: Table
4 shows the Sentinel-2 only and early fusion results, Table 5 displays the
Type-1 late fusion results, and Table 6 presents the Type-2 late fusion
results.

The best mIoU result with the Sentinel-2 only dataset was achieved by training
ResUnet on the Montreal dataset and evaluating it with the Montreal dataset
using the BCE loss function (Table 4). In early fusion experiments, ResUnet
achieved similar and slightly better mIoU results with the focal loss.
However, all results showed a decrease in the mBoundary-IoU metric by a
magnitude of $0.1{\sim}0.01$ compared to the mIoU score for the same
experiment. Furthermore, there was a disagreement between the mIoU and
mBoundary-IoU results when considering different loss functions in the same
model and work area. For example, in the case of early fusion, in the Istanbul
work area, the leading loss function was the focal loss with the mIoU metric,
while it was MSE with the mBoundary-IoU metric. When considering cross work
area evaluation, the results worsened when the training and evaluation work
areas were different. The models trained and tested with the Montreal dataset
achieved better results compared to the Istanbul and Istanbul+Montreal
dataset. Although better results were achieved with models trained on the
Montreal data, their mIoU performance dropped significantly (${\sim}0.2$) when
compared to results of a dataset from another work area. This decrease was
smaller for models trained on the Istanbul+Montreal dataset (${\sim}0.08$) and
minimum for models trained on the Istanbul dataset (${\sim}0.04$).

The best mIoU and mBoundary-IoU results in Type-1 experiments were achieved
with ResUnet using the Montreal training dataset and testing it with the
Montreal data using the MSE loss function and concatenation (0.767 in mIoU,
0.601 in mBoundary-IoU) (Table 5). These results showed a slight improvement
compared to the early fusion experiments. Overall, ResUnet was the leading
model, and MSE was the leading loss function in the majority of the
experiments when other variables were constant. The BCE loss function, when
used with average and maximum fusion, caused a significant decrease in
accuracy when other variables were constant. It is noteworthy that the
multiply fusion method was on par with concatenation or even led in many
experiments, especially when combined with focal loss. The disagreement
observed between mIoU and mBoundary-IoU in Sentinel-2 only and early fusion
experiments persisted. Additionally, the differences observed in the cross
work area evaluation were still present, and these differences were increased
in Type-1 experiments.

In Type-2 experiments, the best mIoU and mBoundary-IoU results were achieved
with ResUnet using the Montreal training dataset and testing it with the
Montreal data using the MSE loss function and concatenation (0.784 in mIoU,
0.631 in mBoundary-IoU) (Table 6). This represents a significant improvement
compared to Type-1 and early fusion experiments. Similar to Type-1, ResUnet
and MSE were the leading model and loss function, respectively. The decreased
performance of BCE with average and maximum fusion methods still persisted,
and the magnitude of accuracy decrease was greater compared to Type-1.
Additionally, the observed disagreements between mIoU and mBoundary-IoU were
still present, and the differences in cross-area evaluation were even more
pronounced.

In the cross work area evaluation, both quantitative and qualitative (Figure
4) evaluations showed that the models’ generalization was limited. However,
the experimental results suggested that early fusion methods were able to
generalize better compared to late fusion alternatives, although early fusion
methods achieved lower mIoU and mBoundary-IoU scores. It was particularly
significant that the models were able to generalize better to wider roads when
GPS trajectory data was fused. Additionally, the models trained with Istanbul
data performed worse in generalization compared to the Montreal data when
other variables were constant. Moreover, the models trained and tested with
Istanbul data achieved lower mIoU and mBoundary-IoU scores compared to the
models trained and tested with Montreal data. The differences in land coverage
propagation and settlement characteristics were considered the main reasons
for this discrepancy. This aspect is further analyzed in Section 4.4 with a
complexity similarity comparison, which examines the variable land coverage
and settlement between the two cities.

Table 4: Baseline experiments with Sentinel RGB-I data. Work Area | Model | Test Area | Sentinel-2 | Sentinel-2+GPS Trajectory
---|---|---|---|---
mIoU | mBoundary IoU | mIoU | mBoundary IoU
BCE | Focal | MSE | BCE | Focal | MSE | BCE | Focal | MSE | BCE | Focal | MSE
Istanbul | D-Linknet | Istanbul | 0.641 | 0.643 | 0.649 | 0.520 | 0.510 | 0.524 | 0.659 | 0.663 | 0.662 | 0.533 | 0.522 | 0.534
Ist.+Mont. | 0.603 | 0.616 | 0.607 | 0.514 | 0.506 | 0.515 | 0.643 | 0.656 | 0.639 | 0.528 | 0.521 | 0.527
Montreal | 0.561 | 0.587 | 0.562 | 0.506 | 0.501 | 0.506 | 0.623 | 0.648 | 0.614 | 0.521 | 0.521 | 0.518
ResUnet | Istanbul | 0.657 | 0.648 | 0.659 | 0.531 | 0.517 | 0.535 | 0.644 | 0.664 | 0.661 | 0.527 | 0.523 | 0.536
Ist.+Mont. | 0.593 | 0.610 | 0.595 | 0.515 | 0.510 | 0.517 | 0.619 | 0.655 | 0.627 | 0.519 | 0.521 | 0.524
Montreal | 0.524 | 0.569 | 0.529 | 0.498 | 0.503 | 0.498 | 0.592 | 0.646 | 0.592 | 0.512 | 0.519 | 0.512
U-Net | Istanbul | 0.634 | 0.630 | 0.636 | 0.518 | 0.508 | 0.520 | 0.645 | 0.661 | 0.654 | 0.525 | 0.521 | 0.528
Ist.+Mont. | 0.581 | 0.610 | 0.582 | 0.507 | 0.505 | 0.507 | 0.623 | 0.659 | 0.635 | 0.519 | 0.524 | 0.523
Montreal | 0.526 | 0.590 | 0.524 | 0.495 | 0.502 | 0.493 | 0.599 | 0.658 | 0.615 | 0.513 | 0.526 | 0.518
Ist.+Mont. | D-Linknet | Istanbul | 0.618 | 0.630 | 0.622 | 0.513 | 0.507 | 0.514 | 0.648 | 0.651 | 0.645 | 0.524 | 0.514 | 0.520
Ist.+Mont. | 0.650 | 0.649 | 0.648 | 0.526 | 0.514 | 0.527 | 0.677 | 0.675 | 0.673 | 0.539 | 0.526 | 0.533
Montreal | 0.680 | 0.664 | 0.668 | 0.538 | 0.521 | 0.537 | 0.702 | 0.694 | 0.699 | 0.553 | 0.537 | 0.545
ResUnet | Istanbul | 0.614 | 0.637 | 0.633 | 0.515 | 0.511 | 0.519 | 0.646 | 0.633 | 0.640 | 0.527 | 0.508 | 0.523
Ist.+Mont. | 0.656 | 0.669 | 0.671 | 0.533 | 0.522 | 0.539 | 0.671 | 0.664 | 0.668 | 0.541 | 0.520 | 0.539
Montreal | 0.691 | 0.698 | 0.706 | 0.549 | 0.533 | 0.559 | 0.691 | 0.695 | 0.691 | 0.554 | 0.533 | 0.553
U-Net | Istanbul | 0.602 | 0.633 | 0.606 | 0.509 | 0.510 | 0.511 | 0.617 | 0.647 | 0.640 | 0.517 | 0.516 | 0.523
Ist.+Mont. | 0.637 | 0.661 | 0.628 | 0.525 | 0.520 | 0.523 | 0.646 | 0.675 | 0.674 | 0.530 | 0.527 | 0.541
Montreal | 0.664 | 0.685 | 0.642 | 0.540 | 0.530 | 0.532 | 0.671 | 0.703 | 0.703 | 0.543 | 0.540 | 0.558
Montreal | D-Linknet | Istanbul | 0.495 | 0.520 | 0.524 | 0.484 | 0.487 | 0.488 | 0.566 | 0.591 | 0.590 | 0.495 | 0.499 | 0.500
Ist.+Mont. | 0.602 | 0.607 | 0.624 | 0.520 | 0.514 | 0.528 | 0.651 | 0.657 | 0.651 | 0.536 | 0.526 | 0.528
Montreal | 0.703 | 0.690 | 0.722 | 0.555 | 0.540 | 0.566 | 0.730 | 0.724 | 0.705 | 0.575 | 0.553 | 0.554
ResUnet | Istanbul | 0.516 | 0.521 | 0.534 | 0.487 | 0.487 | 0.490 | 0.577 | 0.576 | 0.581 | 0.499 | 0.495 | 0.499
Ist.+Mont. | 0.639 | 0.635 | 0.645 | 0.542 | 0.528 | 0.542 | 0.670 | 0.672 | 0.670 | 0.548 | 0.540 | 0.548
Montreal |  0.760 | 0.743 | 0.755 | 0.596 | 0.568 | 0.595 | 0.758 |  0.763 | 0.757 | 0.596 | 0.583 | 0.597
U-Net | Istanbul | 0.505 | 0.517 | 0.486 | 0.486 | 0.487 | 0.480 | 0.598 | 0.602 | 0.591 | 0.502 | 0.500 | 0.499
Ist.+Mont. | 0.627 | 0.626 | 0.597 | 0.535 | 0.521 | 0.523 | 0.675 | 0.674 | 0.670 | 0.543 | 0.531 | 0.543
Montreal | 0.745 | 0.730 | 0.702 | 0.582 | 0.553 | 0.564 | 0.749 | 0.746 | 0.746 | 0.584 | 0.565 | 0.585
|  |  |  |  |  |  |  |  |  |  |  |  |  |

Figure 5: Experiment results of Type-1 late fusion networks.

Figure 6: Experiment results of Type-2 late fusion networks.

(a) Input data: Istanbul test set

(b) Input data: Montreal test set

(c) Output data: Best performing models tested with Istanbul data.

(d) Output data: Best performing models tested with Montreal data.

Figure 4: Generalization capabilities of different model types which are
trained in different dataset: (a) and (b) shows the input data from Istanbul
and Montreal test set respectively. (c) and (d) provides the output of best
performing models - the columns show the dataset which model is trained on,
rows show the type of fusion in use.

Finally, Table 7 provides a summary of the best achieved results and their
comparison to the results of [8], which served as the literature benchmark for
road extraction using Sentinel-2 data. The best models trained with Istanbul
and Montreal data surpassed the mIoU scores reported in the literature,
particularly in the cases of late Type-1 and Type-2 networks, when GPS
trajectory and Sentinel-2 data were utilized. In addition to IoU results
mBoundary-IoU results are also available which can be used as the future
benchmark for shape based comparison.

Table 7: Comparison with benchmarks and our best performing models. Model | IoU | mBoundary IoU*
---|---|---
U-Net + Bicubic x4 |  |
Overall [8] | 0.6894 | -
U-Net + Bicubic x4 |  |
Best [8] | 0.7066 | -
ResUnet Type-2 |  |
with Concatenation & MSE Loss |  |
trained in Istanbul+Montreal |  |
(Best in Istanbul test samples)* | 0.713 | 0.580
ResUnet |  |
without GPS Trajectory fusion |  |
with BCE Loss trained in Montreal* | 0.760 | 0.596
ResUnet Early fusion |  |
with Focal Loss |  |
trained in Montreal* | 0.763 | 0.583
ResUnet Type-1 fusion |  |
with Concatenation & MSE Loss |  |
trained in Istanbul+Montreal* | 0.767 | 0.601
ResUnet Type-2 fusion |  |
with Concatenation & MSE Loss |  |
trained in Istanbul+Montreal* | 0.784 | 0.631
* Our contributions |  |

###  4.4 Complexity analysis

Due to the differences in the metric results of the same models on different
dataset, it is necessary to determine if the two dataset have similar inputs
in terms of complexity. The evaluation results suggest that the complexity of
the Istanbul dataset differs from that of the Montreal dataset, and the models
trained on their respective work areas exhibit varying levels of accuracy. The
complexity of an image dataset can be analyzed by measuring the differences in
entropy [53] or the texture homogeneity derived from the Gray-Level Co-
Occurrence Matrix (GLCM) [54, 55]. Entropy represents the uncertainty of a
system [56], while GLCM is a matrix that illustrates the spatial distribution
of gray levels within an image, providing additional information such as
texture, contrast, and correlation. In this study, entropy (calculated using
[57] and homogeneity from GLCM (calculated using [58] are computed for each
image patch from Istanbul and Montreal, and the distribution of these values
are visualized in Figure 5.

The mean entropy value for Istanbul ($\mu^{e}_{Istanbul}$) is higher than that
of Montreal ($\mu^{e}_{Montreal}$), indicating that, on average, the Istanbul
examples exhibit higher levels of variability compared to the examples from
Montreal ($var^{e}_{Istanbul}\>>\>var^{e}_{Montreal}$). Additionally, the
variability of entropy examples in Istanbul is more diverse than in Montreal.
On the other hand, in terms of homogeneity, the mean value for Montreal
($\mu^{h}_{Montreal}$) examples is higher than that of Istanbul
($\mu^{h}_{Istanbul}$). This implies that the examples in Montreal are more
homogeneous and provide less complex information compared to those in
Istanbul. Similar to entropy, the variability of examples in Istanbul is
higher than the examples in Montreal
($var^{h}_{Istanbul}\>>\>var^{h}_{Montreal}$).

(a) Entropy

(b) Homogeneity

Figure 5: Complexity assessment of training examples from different work
areas: (a) entropy and (b) homogeneity distribution in Istanbul and Montreal.

##  5 Discussion and Concluding Remarks

In this study, the performance of different deep learning models, loss
functions, fusion approaches, and model generalization is evaluated for road
extraction tasks using low-resolution satellite imagery and GPS trajectory
data. The evaluation of the results is conducted using a region-based metric
and a shape-based metric, with using a new benchmark dataset covering Istanbul
and Montreal. The results indicate that ResUnet outperforms U-Net and
D-Linknet in road extraction tasks and achieves better results than the
benchmark study by [8] using low-resolution Sentinel-2 data.

Overall, the performance of road extraction results improves when GPS
trajectory data is fused with satellite imagery, particularly in the case of
late fusion Type-2 with concatenation and multiply methods. Among the
evaluated loss functions, MSE performs the best, while focal loss and BCE
perform slightly worse, with BCE demonstrating a significant drop in
performance when used in combination with average and maximum fusion methods.
Additionally, the evaluation metrics provide novel insights into road
extraction. The shape-based mBoundary-IoU metric generally provides similar
information to the region-based IoU metric, although there are instances of
disagreement, indicating that IoU may not be always reliable considering the
shape of the output.

Regarding model generalization, the consistency of results among different
models suggests that early fusion performs better while cross work area when
testing compared to Type-1 and Type-2 late fusion networks.

In addition to the above findings, an analysis is conducted to understand the
performance differences when training on different work areas. This analysis
evaluates the complexity of the Istanbul and Montreal datasets using entropy
and homogeneity measures, and concludes that the Istanbul dataset is more
complex compared to the Montreal dataset.

It is worth noting that in the field of semantic segmentation, there are more
complex models available recently, including Transformer-based models.
Additionally, other loss functions, regularization strategies, and fusion
methods can be considered to further extend the findings of this study. Beyond
the ablation studies reported in this paper, further exploration of such
architectural engineering approaches is kept outside the scope of this paper.
Moreover, the complexity analysis carried out in this study can be expanded
with additional complexity measures and can be used as an additional factor
for the models.

Acknowledgement This preprint has not undergone peer review (when applicable)
or any post-submission improvements or corrections. The Version of Record of
this article is published in Earth Science Informatics [ESIN], and is
available online at https://doi.org/10.1007/s12145-023-01201-6.

The numerical calculations reported in this paper were fully performed at
TUBITAK ULAKBIM, High Performance and Grid Computing Center (TRUBA resources).
Authors would like to thank Istanbul Metropolitan Municipality and City of
Montreal for GPS trajectory dataset, European Space Agency (ESA) for
Sentinel-2 data and OpenStreetMap Foundation and OpenStreetMap Contributers
for OpenStreetMap data. This study is part of the Ph.D thesis conducted in
Istanbul Technical University by the first author. Authors would like to thank
the Ph.D thesis advancement monitoring committee members, Gulsen Kaya Taskin
and Taskin Kavzoglu.

Code & data availability statement The code of the experiments and the data
used in the experiments of this study made available online and related
website information shared within the article.

## References

  *   * Jiao et al. [2021] Jiao, C., Heitzler, M., Hurni, L.: A survey of road feature extraction methods from raster maps. Transactions in GIS 25(6), 2734–2763 (2021)
  * Zhou et al. [2018] Zhou, L., Zhang, C., Wu, M.: D-linknet: Linknet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction, vol. 2018-June, pp. 192–196. IEEE Computer Society, Salt Lake City (2018). https://doi.org/10.1109/CVPRW.2018.00034
  * Demir et al. [2018] Demir, I., Koperski, K., Lindenbaum, D., Pang, G., Huang, J., Basu, S., Hughes, F., Tuia, D., Raskar, R.: Deepglobe 2018: A challenge to parse the earth through satellite images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 172–17209 (2018). https://doi.org/10.1109/CVPRW.2018.00031
  * Liu et al. [2022] Liu, P., Wang, Q., Yang, G., Li, L., Zhang, H.: Survey of road extraction methods in remote sensing images based on deep learning. PFG – Journal of Photogrammetry, Remote Sensing and Geoinformation Science 90(2), 135–159 (2022) https://doi.org/10.1007/s41064-022-00194-z
  * Mnih and Hinton [2010] Mnih, V., Hinton, G.E.: Learning to detect roads in high-resolution aerial images. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) Computer Vision – ECCV 2010, pp. 210–223. Springer, Berlin, Heidelberg (2010)
  * Zhang et al. [2018] Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual U-Net. IEEE Geoscience and Remote Sensing Letters 15(5), 749–753 (2018) https://doi.org/10.1109/LGRS.2018.2802944
  * Sun et al. [2018] Sun, T., Chen, Z., Yang, W., Wang, Y.: Stacked U-Nets with multi-output for road extraction. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 187–1874 (2018). https://doi.org/10.1109/CVPRW.2018.00033
  * Ayala et al. [2021] Ayala, C., Aranda, C., Galar, M.: Towards fine-grained road maps extraction using Sentinel-2 imagery. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences V-3-2021, 9–14 (2021) https://doi.org/10.5194/isprs-annals-V-3-2021-9-2021
  * Johnson et al. [2022] Johnson, N., Treible, W., Crispell, D.: OpenSentinelMap: A large-scale land use dataset using OpenStreetMap and Sentinel-2 imagery. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1332–1340 (2022). https://doi.org/10.1109/CVPRW56347.2022.00139
  * Edelkamp and Schrödl [2003] Edelkamp, S., Schrödl, S.: In: Klein, R., Six, H.-W., Wegner, L. (eds.) Route Planning and Map Inference with Global Positioning Traces, pp. 128–151. Springer, Berlin, Heidelberg (2003)
  * Biagioni and Eriksson [2012] Biagioni, J., Eriksson, J.: Map inference in the face of noise and disparity. In: Proceedings of the 20th International Conference on Advances in Geographic Information Systems. SIGSPATIAL ’12, pp. 79–88. Association for Computing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2424321.2424333 . https://doi.org/10.1145/2424321.2424333
  * Karagiorgou and Pfoser [2012] Karagiorgou, S., Pfoser, D.: On vehicle tracking data-based road network generation. In: Proceedings of the 20th International Conference on Advances in Geographic Information Systems. SIGSPATIAL ’12, pp. 89–98. Association for Computing Machinery, New York, NY, USA (2012). https://doi.org/10.1145/2424321.2424334 . https://doi.org/10.1145/2424321.2424334
  * Tang et al. [2019] Tang, J., Deng, M., Huang, J., Liu, H., Chen, X.: An automatic method for detection and update of additive changes in road network with gps trajectory data. ISPRS International Journal of Geo-Information 8(9) (2019) https://doi.org/10.3390/ijgi8090411
  * Sun et al. [2019] Sun, T., Di, Z., Che, P., Liu, C., Wang, Y.: Leveraging crowdsourced gps data for road extraction from aerial imagery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)
  * Liu et al. [2022] Liu, L., Yang, Z., Li, G., Wang, K., Chen, T., Lin, L.: Aerial images meet crowdsourced trajectories: A new approach to robust road extraction. IEEE Transactions on Neural Networks and Learning Systems, 1–15 (2022) https://doi.org/10.1109/TNNLS.2022.3141821
  * Wu et al. [2020] Wu, H., Zhang, H., Zhang, X., Sun, W., Zheng, B., Jiang, Y.: DeepDualMapper: A gated fusion network for automatic map extraction using aerial images and trajectories. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 1037–1045 (2020)
  * Yang et al. [2022] Yang, J., Ye, X., Wu, B., Gu, Y., Wang, Z., Xia, D., Huang, J.: DuARE: Automatic road extraction with aerial images and trajectory data at baidu maps. In: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. KDD ’22, pp. 4321–4331. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3534678.3539029 . https://doi.org/10.1145/3534678.3539029
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp. 234–241. Springer, Cham (2015)
  * Wu et al. [2022] Wu, Z., Zhang, J., Zhang, L., Liu, X., Qiao, H.: Bi-HRNet: A road extraction framework from satellite imagery based on node heatmap and bidirectional connectivity. Remote Sensing 14(7) (2022) https://doi.org/10.3390/rs14071732
  * Xie et al. [2019] Xie, Y., Miao, F., Zhou, K., Peng, J.: HsgNet: A road extraction network based on global perception of high-order spatial information. ISPRS International Journal of Geo-Information 8(12) (2019) https://doi.org/10.3390/ijgi8120571
  * Dai et al. [2023] Dai, L., Zhang, G., Zhang, R.: RADANet: Road augmented deformable attention network for road extraction from complex high-resolution remote-sensing images. IEEE Transactions on Geoscience and Remote Sensing 61, 1–13 (2023) https://doi.org/10.1109/TGRS.2023.3237561
  * Yang et al. [2022] Yang, M., Yuan, Y., Liu, G.: SDUNet: Road extraction via spatial enhanced and densely connected unet. Pattern Recognition 126, 108549 (2022) https://doi.org/10.1016/j.patcog.2022.108549
  * Li et al. [2022] Li, S., Liao, C., Ding, Y., Hu, H., Jia, Y., Chen, M., Xu, B., Ge, X., Liu, T., Wu, D.: Cascaded residual attention enhanced road extraction from remote sensing images. ISPRS International Journal of Geo-Information 11(1) (2022) https://doi.org/10.3390/ijgi11010009
  * Luo et al. [2022] Luo, L., Wang, J.-X., Chen, S.-B., Tang, J., Luo, B.: BDTNet: Road extraction by bi-direction transformer from remote sensing images. IEEE Geoscience and Remote Sensing Letters 19, 1–5 (2022) https://doi.org/10.1109/LGRS.2022.3183828
  * Jiang et al. [2022] Jiang, X., Li, Y., Jiang, T., Xie, J., Wu, Y., Cai, Q., Jiang, J., Xu, J., Zhang, H.: RoadFormer: Pyramidal deformable vision transformers for road network extraction with remote sensing images. International Journal of Applied Earth Observation and Geoinformation 113, 102987 (2022) https://doi.org/10.1016/j.jag.2022.102987
  * Tao et al. [2023] Tao, J., Chen, Z., Sun, Z., Guo, H., Leng, B., Yu, Z., Wang, Y., He, Z., Lei, X., Yang, J.: Seg-Road: A segmentation network for road extraction based on transformer and cnn with connectivity structures. Remote Sensing 15(6) (2023) https://doi.org/10.3390/rs15061602
  * Ayala et al. [2021] Ayala, C., Sesma, R., Aranda, C., Galar, M.: A deep learning approach to an enhanced building footprint and road detection in high-resolution satellite imagery. Remote Sensing 13(16) (2021) https://doi.org/10.3390/rs13163135
  * Li et al. [2021] Li, P., He, X., Qiao, M., Miao, D., Cheng, X., Song, D., Chen, M., Li, J., Zhou, T., Guo, X., Yan, X., Tian, Z.: Exploring multiple crowdsourced data to learn deep convolutional neural networks for road extraction. International Journal of Applied Earth Observation and Geoinformation 104, 102544 (2021) https://doi.org/10.1016/j.jag.2021.102544
  * Zhou et al. [2021] Zhou, K., Xie, Y., Gao, Z., Miao, F., Zhang, L.: Funet: A novel road extraction network with fusion of location data and remote sensing imagery. ISPRS International Journal of Geo-Information 10(1) (2021) https://doi.org/10.3390/ijgi10010039
  * Eftelioglu et al. [2022] Eftelioglu, E., Garg, R., Kango, V., Gohil, C., Chowdhury, A.R.: RING-Net: Road inference from gps trajectories using a deep segmentation network. In: Proceedings of the 10th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data. BigSpatial ’22, pp. 17–26. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3557917.3567617 . https://doi.org/10.1145/3557917.3567617
  * Gao et al. [2021] Gao, L., Wang, J., Wang, Q., Shi, W., Zheng, J., Gan, H., Lv, Z., Qiao, H.: Road extraction using a dual attention dilated-linknet based on satellite images and floating vehicle trajectory data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 14, 10428–10438 (2021) https://doi.org/10.1109/JSTARS.2021.3116281
  * Zhang et al. [2021] Zhang, Y., Sidibé, D., Morel, O., Mériaudeau, F.: Deep multimodal fusion for semantic image segmentation: A survey. Image and Vision Computing 105, 104042 (2021) https://doi.org/10.1016/j.imavis.2020.104042
  * Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org
  * Maxwell et al. [2021a] Maxwell, A.E., Warner, T.A., Guillén, L.A.: Accuracy assessment in convolutional neural network-based deep learning remote sensing studies—part 1: Literature review. Remote Sensing 13(13) (2021) https://doi.org/10.3390/rs13132450
  * Maxwell et al. [2021b] Maxwell, A.E., Warner, T.A., Guillén, L.A.: Accuracy assessment in convolutional neural network-based deep learning remote sensing studies—part 2: Recommendations and best practices. Remote Sensing 13(13) (2021) https://doi.org/10.3390/rs13132591
  * Etten [2020] Etten, A.V.: City-scale road extraction from satellite imagery v2: Road speeds and travel times. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, Snowmass Village (2020). https://doi.org/10.1109/wacv45572.2020.9093593 . https://doi.org/10.1109/wacv45572.2020.9093593
  * Etten et al. [2020] Etten, A.V., Shermeyer, J., Hogan, D., Weir, N., Lewis, R.: Road network and travel time extraction from multiple look angles with spacenet data. In: IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium. IEEE, Virtual Symposium (2020). https://doi.org/10.1109/igarss39084.2020.9324091 . https://doi.org/10.1109/igarss39084.2020.9324091
  * Maier-Hein et al. [2023] Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M.D., Büttner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., Reyes, M., Riegler, M.A., Wiesenfarth, M., Kavur, E., Sudre, C.H., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Rädsch, A.T., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Blaschko, M., Cardoso, M.J., Cheplygina, V., Cimini, B.A., Collins, G.S., Farahani, K., Ferrer, L., Galdran, A., Ginneken, B., Haase, R., Hashimoto, D.A., Hoffman, M.M., Huisman, M., Jannin, P., Kahn, C.E., Kainmueller, D., Kainz, B., Karargyris, A., Karthikesalingam, A., Kenngott, H., Kofler, F., Kopp-Schneider, A., Kreshuk, A., Kurc, T., Landman, B.A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A.L., Mattson, P., Meijering, E., Menze, B., Moons, K.G.M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rajpoot, N., Rieke, N., Saez-Rodriguez, J., Sánchez, C.I., Shetty, S., Smeden, M., Summers, R.M., Taha, A.A., Tiulpin, A., Tsaftaris, S.A., Calster, B.V., Varoquaux, G., Jäger, P.F.: Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv e-prints (2023) arXiv:2206.01653 [cs.CV]
  * Ozturk et al. [2022] Ozturk, O., Isik, M.S., Sariturk, B., Seker, D.Z.: Generation of istanbul road data set using google map api for deep learning-based segmentation. International Journal of Remote Sensing 43, 2793–2812 (2022) https://doi.org/10.1080/01431161.2022.2068989
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * Ciampiconi et al. [2023] Ciampiconi, L., Elwood, A., Leonardi, M., Mohamed, A., Rozza, A.: A survey and taxonomy of loss functions in machine learning (2023)
  * Jadon [2020] Jadon, S.: A survey of loss functions for semantic segmentation. In: 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB), pp. 1–7 (2020). https://doi.org/10.1109/CIBCB48159.2020.9277638
  * Lin et al. [2020] Lin, T.-Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(2), 318–327 (2020) https://doi.org/10.1109/TPAMI.2018.2858826
  * Cheng et al. [2021] Cheng, B., Girshick, R., Dollar, P., Berg, A.C., Kirillov, A.: Boundary iou: Improving object-centric image segmentation evaluation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15334–15342 (2021)
  * Istanbul Buyuksehir Belediyesi [2020] Istanbul Buyuksehir Belediyesi: IBB ISTAÇ Araçlarının Anlık Konum ve Hız Bilgileri . https://data.ibb.gov.tr/dataset/ibb-istac-araclarinin-anlik-konum-ve-hiz-bilgileri. [Accessed 23-Jun-2020] (2020)
  * Ville de Montréal [2021] Ville de Montréal: VMTL-MTL-Trajet. https://www.donneesquebec.ca/recherche/fr/dataset/vmtl-mtl-trajet. [Accessed 12-Apr-2023] (2021)
  * European Space Agency [2021] European Space Agency: Sentinel-2 MSI - MultiSpectral Instrument User Guide. https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-2-msi. [Accessed 12-Apr-2023] (2021)
  * OpenStreetMap Contributors [2021] OpenStreetMap Contributors: OpenStreetMap. https://www.openstreetmap.org/. [Accessed 12-Apr-2023] (2021)
  * Yuan et al. [2019] Yuan, M., Liu, Z., Wang, F., Jin, F.: Rethinking labelling in road segmentation. International Journal of Remote Sensing 40(22), 8359–8378 (2019) https://doi.org/10.1080/01431161.2019.1608393
  * Gengec and Tari [2021] Gengec, N.E., Tari, E.: Performance evaluation of gps trajectory rasterization methods. In: Computational Science and Its Applications – ICCSA 2021: 21st International Conference, Cagliari, Italy, September 13–16, 2021, Proceedings, Part I, pp. 3–17. Springer, Berlin, Heidelberg (2021). https://doi.org/10.1007/978-3-030-86653-2_1 . {}{}}{https://doi.org/10.1007/978-3-030-86653-2_1}{cmtt}
  * Abadi et al. [2015] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Software available from tensorflow.org (2015). https://www.tensorflow.org/
  * TensorFlow Addons Contributors [2021] TensorFlow Addons Contributors: TensorFlow Addons. https://github.com/tensorflow/addons
  * Perkiö and Hyvärinen [2009] Perkiö, J., Hyvärinen, A.: Modelling image complexity by independent component analysis, with application to content-based image retrieval. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds.) Artificial Neural Networks – ICANN 2009, pp. 704–714. Springer, Berlin, Heidelberg (2009)
  * Haralick et al. [1973] Haralick, R.M., Shanmugam, K., Dinstein, I.: Textural features for image classification. IEEE Transactions on Systems, Man, and Cybernetics SMC-3(6), 610–621 (1973) https://doi.org/10.1109/TSMC.1973.4309314
  * Rahane and Subramanian [2020] Rahane, A.A., Subramanian, A.: Measures of complexity for large scale image datasets. In: 2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pp. 282–287 (2020). https://doi.org/10.1109/ICAIIC48513.2020.9065274
  * Bilgi et al. [2019] Bilgi, S., Gulnerman, A.G., Arslanoglu, B., Karaman, H., Ozturk, O.: Complexity measures of sports facilities allocation in urban area by metric entropy and public demand compatibility. International Journal of Engineering and Geosciences 4(3), 141–148 (2019) https://doi.org/10.26833/ijeg.540180
  * scikit-image [2021a] scikit-image: Shannon entropy. https://scikit-image.org/docs/stable/api/skimage.measure.html#skimage.measure.shannon_entropy. [Online; accessed 12-Apr-2023] (2021)
  * scikit-image [2021b] scikit-image: Gray-level co-occurrence matrix properties. https://scikit-image.org/docs/stable/api/skimage.feature.html#skimage.feature.graycoprops. [Online; accessed 12-Apr-2023] (2021)