11institutetext: IMT School for Advanced Studies, Lucca, Italy
11email: {zainab.alwaisi, simone.soderi<EMAIL_ADDRESS>22institutetext: CINI Cybersecurity Laboratory, Roma, Italy

# Mitigating and Analysis of
Memory Usage Attack in IoE system

Anonymous  Zainab Alwaisi 11 Simone Soderi 1122 Rocco De Nicola 1122

###### Abstract

Internet of Everything (IoE) is a newly emerging trend, especially in homes.
Marketing forces toward smart homes are also accelerating the spread of IoE
devices in households. An obvious risk of the rapid adoption of these smart
devices is that many lack controls for protecting the privacy and security of
end users from attacks designed to disrupt lives and incur financial losses.
Today the smart home is a system for managing the basic life support processes
of both small systems, e.g., commercial, office premises, apartments,
cottages, and largely automated complexes, e.g., commercial and industrial
complexes. One of the critical tasks to be solved by the concept of a modern
smart home is the problem of preventing the usage of IoE resources. Recently,
there has been a rapid increase in attacks on consumer IoE devices.

Memory corruption vulnerabilities constitute a significant class of
vulnerabilities in software security through which attackers can gain control
of an entire system. Numerous memory corruption vulnerabilities have been
found in IoE firmware already deployed in the consumer market. This paper aims
to analyze and explain the resource usage attack and create a low-cost
simulation environment to aid in the dynamic analysis of the attack. Further,
we perform controlled resource usage attacks while measuring resource
consumption on resource-constrained victims’ IoE devices, such as CPU and
memory utilization. We also build a lightweight algorithm to detect memory
usage attacks in the IoE environment. The result shows high efficiency in
detecting and mitigating memory usage attacks by detecting when the intruder
starts and stops the attack.

###### Keywords:

Smart Home (SH) Internet of Everything (IoE) memory usage attack detection
security resource constraint.

##  1 Introduction

The Internet of Everything (IoE) encompasses data, people, the Internet of
Things (IoT), and processes. IoE builds on IoT, which focuses on connecting
network devices equipped with specialized sensors or actuators through the
Internet [5]. The sensors and actuators can detect and respond to
environmental changes, including light, temperature, sound, vibration, etc.
IoE dramatically expands the scope of IoT by adding components that can
provide richer experiences for businesses, individuals, and countries. For
example, instead of simply relying on things to interact with their
environments, as shown in Figure 1, IoE can leverage all related data and
processes to make IoT more relevant and valuable to people [19]. The ultimate
goal of IoE is to boost operational efficiency, offer new business
opportunities, and improve the quality of our lives. Better to relate to this
idea; take the scenario of a person uncertain about closing a gas valve at
home. An IoE solution allows a user to automatically check the gas valve’s
status and close it remotely if necessary [13] [14].

Despite its potential rewards, IoE could pose significant security threats to
its adopters. The number of IoE devices around us is steadily increasing, and
IoE is starting to play a more critical role in our everyday lives. In
particular, the link between the physical world and cyberspace established by
IoE increases the risk of cyber attacks targeting smart devices since attacks
against IoE can directly impact the health [11] and the welfare of their end
users. Building on our gas value scenario, one can easily imagine a threat
scenario in which an attacker causes a gas leak on purpose [20].

Figure 1: The Definition of Internet of Everything (IoE).

Even more alarming is that we are often oblivious to the quantity and nature
of the IoE devices surrounding us, not to mention the potential security risks
they represent. The recent security incidents resulting from IoE security
vulnerabilities corroborate this observation. In particular, one of them is a
Distributed Denial of Service (DDoS) attack against Dyn [2] in October 2016.
This incident involved a botnet called Mirai, consisting of approximately
$100,000$ IoE hosts, including digital cameras and routers. The Mirai botnet
launched DDoS attacks against Dyn, bringing down its Domain Name Servers
(DNS), which resulted in an outage of major commercial websites, e.g., Netflix
and CNN [7]. Another attack could affect the sensors and actuators’ resources,
making the smart devices unavailable to end-users.

Due to these emerging threats, it is imperative to raise awareness of
potential IoE security risks [17] among end users through systematic risk
assessment and effective visualizations. Home users are especially vulnerable
because they are increasingly surrounded by IoE appliances, e.g., hands-free
speakers, baby monitors, and security cameras. However, they lack the
resources and skills to identify their IoE-related threats, remediate them,
and minimize potential security risks. Therefore, in this paper, we mainly
focus on analyzing the memory usage attack in smart devices and mitigating the
effect of this attack by building a lightweight algorithm to detect memory
usage attacks by calculating the memory usage of the smart device.

To accomplish this goal in home networks, we first identify memory usage
attacks in smart home devices. Next, we analyze the effect of the attack by
sending malicious attacks to affect the resources of the smart devices and
calculate its effect on memory usage. We then elicit and document threats in
the form of threat scenarios. Once specified, we build a lightweight algorithm
to detect and mitigate the effect of the attack on memory usage.

###  1.1 Motivation and Contribution

IoE is a fast-growing field with capabilities to revolutionize the whole
industry. As per market trends, more than $20$ billion smart devices will be
deployed in the next five years [10]. These interconnected devices will be
generating sensitive data which needs to be protected. The field of IoE is
making leaps and bounds technologically. There are multiple limitations while
deploying IoE devices daily, e.g., battery life and lightweight computation.
Therefore, building a novel security mechanism aims to protect the
functionalities and privacy of sensitive IoE network environments, including
healthcare, smart cities, etc. However, due to the substantial number of nodes
in the environment and their restricted computing capabilities, securing smart
nodes in the IoE environment is essential to protect the data and make the
devices available to end-users. A lightweight mitigation technique should be
considered to protect smart devices from resource-constraint attacks such as
Denial of Services (DoS), Distributed Denial of Services (DDoS), and other
malicious attacks. Our main contribution is building a lightweight technique
to detect memory usage attacks in smart devices deployed directly at sensors.
It applies real-time memory usage calculation to discriminate between
different memory usage, e.g., read/write to memory. In this work, we consider
different behaviors on the memory of smart devices. We measure the memory
usage when there is read and write, under or without the attack, to evaluate
the best detection of memory usage attack. We simulate the mitigation
technique and assess the results by applying the proposed technique to smart
devices, such as the Raspberry Pi111https://www.raspberrypi.com/documentation/
and Arduino. We measure the current memory usage of the smart device to
monitor the memory usage to discriminate between normal and abnormal
behaviors. Therefore, this algorithm design is a protection strategy for IoE
devices to maintain their integrity, seamlessly make them available to
legitimate users, and protect them from memory attacks by considering their
resource constraints.

###  1.2 Organization of the paper

We organized this paper as follows. Section 2 presents a related work and
background reading of resource-constrained attacks in IoT systems, e.g.,
memory usage attacks. We discuss the threat scenario and its analysis in
Sections 4 and 5. We describe our proposal, including metrics definition,
methodology, and the detection algorithm, in Section 6.1. In Section 7, we
show the results and discussions. Finally, Section 8 presents some concluding
remarks and future works.

Table 1: Memory Usage analysis before and after the attack.  Device |  Status |  % CPU Usage |  % Memory Usage |
---|---|---|---|---
Raspberry Pi |  Idle | $0.55\div 0.88$ | $10\div 20$ |
|  Active | $0.88\div 1.50$ | $20\div 35$ |
|  Under Attack | $1.5\div 16.5$ | $36\div 66$ |
Device |  Status |  Thread Time [s] |  % Memory Usage |
Arduino |  Idle | $1\div 20$ | $8\div 11$ |
|  Active | $21\div 45$ | $11\div 16$ |
|  Under Attack | $\geq 45$ | $17\div 45$ |

##  2 Related Work

The IoE links people, data, things, and processes to make interconnections
easier and more far-reaching than ever before [6]. As such, everyday
appliances should be subjected to rigorous cyber security testing to the same
degree that these appliances are tested and measured for traditional
qualities, e.g., durability, fit-for-purpose, maintenance, etc. Unfortunately,
standardized and independent verification of IoE devices is in its nascent
stage, with IoE security being the focus of legislation and standard security
criteria. Different authors tried to mitigate and detect attacks by analyzing
the memory. Memory analysis has attracted several malware researchers. Vömel
_et al._ in [16] surveyed the main memory acquisition and analysis techniques.
In [12], Rathnayaka, et al. have observed that successful malware infection
leaves a memory footprint. Zaki _et al._ in [18] studied the artifacts left by
rootkits at the kernel level, such as driver, module, System Service Dispatch
Table (SSDT) hook, Interrupt Descriptor Table (IDT) hook, and callback. The
experiments proved that certain activities, such as callback functions,
modified drivers, and attached devices, are the most suspicious activities at
the kernel level. In [1], Aghaeikheirabady presented an analysis approach that
extracts features available in memory, such as function calls, Dynamic-Link
Libraries (DLLs), and registry, and compares the information available in
different memory structures to increase the accuracy. The approach relies on
the frequencies of the extracted features to classify them, and an overall
accuracy of $98$% is measured by applying Naïve Bayes. However, a significant
drawback is the high False Positive Rate (FPR) that exceeded $16$%.

Similarly, in [8], Mosli _et al._ introduced a technique that detects malware
based on extracting three features from memory images; API calls, registry,
and imported libraries. However, the experiments were performed on each
feature individually, and maximum accuracy of $96$% was achieved using the
Support Vector Machines (SVM) classifier on the registry activities feature.
Afterward, in their following work [9], Mosli _et al._ utilized the process
handles available in memory to detect malware. The experiment has found that
the handles used by malware are process handles, mutants, and section handles.
However, when applying the random forest classifier, their approach achieved a
modest accuracy slightly higher than $91$%. Likewise, Duan _et al._ [4]
presented an approach to extract live DLL is featured from memory and employed
to detect malware variants that use the same DLLs. The experimental result
showed an accuracy of $90$% achieved using the hidden naïve bayes classifier.
Furthermore, Dai _et al._ [3] proposed a malware detection and classification
approach based on extracting memory images and converting them into fixed-size
greyscale images. The approach then extracted the features from the images,
using a gradient histogram, and used them to classify malware. An accuracy of
$95.2$% was obtained using the neural network classifier. Moreover, the
authors of this work previously combined API calls from behavior analysis and
memory analysis into one vector to represent each sample. A dataset was used,
which consisted of $1200$ malware and $400$ benign files, to train the SVM
classifier. The work confirmed that memory analysis could overcome the
limitations of behavior analysis [15].

In this paper, we monitor the memory usage of the smart devices in the IoE
environment to detect memory usage attacks and mitigate resource-constraints
attacks. We also perform a testbed environment to measure the memory usage of
the smart devices before and after attacking the IoE environment. In the
experiments, the effectiveness of the proposed approach on memory usage attack
detection and classification has been demonstrated and measured by three
evaluations: classification accuracy, monitoring the memory usage, and
detecting the attack. To the best of our knowledge, this is the first work to
examine and detect such kind of memory usage attack in IoE systems.

##  3 Testbed Scenario

We used Raspberry Pi and Arduino as smart home devices in this experiment. We
used different software tools for attacking data generation and collection. On
the adversary side, we used _Nmap_ 222https://nmap.org/ to launch a network
scan and identify devices’ status, such as online or offline, IP address, and
MAC address. Different tools can generate malicious attacks on the victim
side, such as _hping3_ 333https://www.kali.org/tools/hping3/. We used _tshark_
tool 444https://www.wireshark.org/ to evaluate the impact of memory usage
attacks on smart devices and capture WiFi traffic.

We also created a module inside the smart device to monitor memory usage and
register all memory behaviors in the database (DB). The monitoring mode
registers the behavior of the smart devices once it is _Idle_ , _active_ , and
_under attack_.

Figure 2: Testbed scenario showing the devices used in our experiment.

Different stages are used to run our experiment. In the first stage, we
monitor the memory usage once the device is Idle, Active, and under attack.
Then, we run a network scan to capture the port and device status. Once we
ensure that the device is connected to the Internet, we send memory usage
attacks for two purposes: first, to affect the memory, and second, to consume
more memory usage and study the behavior of the attack. Then, we run memory
usage monitoring to calculate the memory usage of the devices and study the
devices’ behaviors before and after the attack.

Figure 3: Testing Environment.

##  4 Threat Scenarios and Threat Model

This section will briefly discuss the design of memory usage attacks on IoE
smart devices. The memory usage attack aims to affect the smart device’s
memory by sending malicious attacks such as DoS or DDoS attacks. In
particular, memory usage attack targets a specific type of vulnerable IoE and
embedded devices because these devices have very little build-in-security
protection and suffer from resource-constraints problems.

###  4.1 Threat Scenario

The smart devices of IoE suffer from low computation problems such as low
energy and memory. The resource-constraints problems encourage attackers to
attack these devices by flooding the smart devices with malicious attacks. In
this paper, we assume that the attacker has gained access to the control
network and can communicate with the smart devices as an insider threat (e.g.,
a consumer who uses current or past authorized access to the smart devices to
exceed or misuse) or an external hacker. A wide range of attacks, such as
malicious attacks, will be available for the attacker when he/she gains access
to a control network. In this paper, we study memory usage attacks on the
smart devices of IoE systems.

The threat scenario used in our experiment was first to scan the network and
get different information about the port and devices’ status. For scanning the
network, we install _Nmap_ on Kali-Linux. In this scenario, the attacker can
send a malicious attack to the smart device to affect its resources in terms
of memory. The IP address, port, and device status are stored in the DB for
further calculation. After scanning the network, we start the monitoring mode
of the smart device’s memory usage; once the device is Idle and active, and
when we send a malicious attack using _hping3_ tools to the smart device. In
this case, we study the memory behavior before and after attacking the smart
devices. We also store all information about the memory, such as memory in
total, memory usage, and CPU usage before and after attacking the smart
devices.

In particular, the source code consists of three different parts:

  * •

Memory usage attack: This module commences with the DoS and DDoS attack to
send malicious attacks to the smart devices and affect their memory.

  * •

Scanner: This module scans the network and gets different information about
the smart devices. Also, it sends the IP address of the attacked smart devices
for further calculation

  * •

Memory-monitoring-mode: This module monitors the memory usage of the smart
devices when it is Idle, active, and under attack. The monitoring mode helps
to register different memory behaviors for detecting such attacks.

###  4.2 Threat Model

We present a model of attacks on the memory usage of the smart devices of IoE
systems, which can be used to understand the possible attack vectors
intuitively and concisely. Also, to build lightweight algorithm
555https://github.com/developerZA/MitigationMemoryAttack.git for detecting
smart devices from these attacks. Let us assume $ATK$ denotes the attacker
while $d$ denotes the smart devices, and $MEM$ denotes the smart devices’
memory usage. According to our model, every attack originates from an attacker
$ATK$ where $atk\in ATK$ by a means towards a target $d$. We can model this
relationship as follows:

$ATK\mapsto mem\to D$ (1)

where $atk\in ATK,d\subset D$, $mem\in MEM$. The notation $\mapsto$ maps the
attacker ($ATK$) to the victim’s ($D$) memory ($mem$).

Figure 4: Schematic diagram of the proposed method.

For calculating the memory usage and CPU usage of the smart devices before and
after attacking the memory, the following equation math represents this
calculation. Let us describe the memory usage measurement ($MEM$) footprints
considering the set of different device statuses in the attack’s absence or
presence.

$MEM(d)=f(mem(d),ATK,n)\quad\textrm{and}\quad n\in[0,1]$ (2)

Where ($mem_{d}$) the memory usage measurement ($mem$) of the smart device
($d$) at a point in time in the absence or presence of cyberattacks for a
specific attack ($ATK$), and $n$ is the number of memory usage measurements in
a time interval, $f(mem(d),ATK,n)\in[0,1]$ where $0$ is the minimum memory
usage measurement, and $1$ presents the maximum memory usage measurement in
the absence or presence of the attack. The CPU ($CPU$) usage measurement is
also calculated for the Raspberry Pi device as follows:

$CPU(d)=f(cpu(d),ATK,n)\quad\textrm{and}\quad n\in[0,1]$ (3)

Where ($cpu_{d}$) is the CPU usage measurement ($cpu$) of the smart device
($d$) at a point in time in the absence or presence of cyberattacks for a
specific attack ($ATK$), and $n$ is the number of CPU usage measurements in a
time interval, $f(cpu(d),ATK,n)\in[0,1]$ where $0$ is the minimum CPU usage
measurement, and $1$ presents the maximum CPU usage measurement in the absence
or presence of the attack.

We do not calculate the CPU usage for the Arduino, as it is a microcontroller.
We focus only on the maximum memory usage through or without the attack using
a particular library called _MemoryFree_ and _pgmStrToRAM_. And we also
calculate _micros()_ or _millis()_ before and after sending the malicious
attack. We also calculate the thread time for different statuses of the smart
device, e.g., Idle, Active, and under attack.

##  5 Static Analysis of Resource Usage Attack

The smart devices used in this experiment were infected with

Figure 5: Raspberry Pi (Memory Usage Before and After the attack).

malicious software used to carry out different malicious attacks on a target
on an isolated network. During the experiments, the memory usage footprints of
the smart devices were obtained under normal operating conditions, as well as
when these smart devices carry out cyberattacks. Each memory usage footprint
was obtained by taking measurements after $5$ s within $1$ minute when the
smart device performs an attack and normal operation. A total of $10$ minutes
of calculation measurement of memory usage footprints of both in the presence
of attacks and normal functioning smart devices were built.

During packet collection, the attacks are sent using the same Transmission
Control Protocol (TCP) and User Datagram Protocol (UDP) flood commands. Using
the topology as depicted by Figure 3, the malicious TCP and UDP traffic are
separately sent to the victim device, while all usage statistics are recorded
on the victim device. Each attack is simulated for a duration of $1$ minute,
and all usage statistics are recorded for the same duration. No attacks are
sent during the first period ($10$ minutes), and all usage statistics are
recorded and saved in the DB. And also, the same things applied once the
second period started after sending malicious attacks.

The result of this experiment shows the memory usage footprint when the device
is Idle, Active, and under attack. Therefore, the normal usage of the memory
of the Raspberry Pi device in the absence of the attack fluctuates between
$10$% to $36$%. This percentage is divided between two different states of the
smart device when it is Idle, the percentage is between $10$ and $20$%, and
when it is Active, the percentage is more than $25$% but less than $37$% as
shown in Figure 5. Moreover, the percentage of memory usage changed after
sending the malicious attack, and the percentage changed to be more than $66$%
per minute. We also calculate the CPU usage of the smart devices to check the
CPU status before and after attacking the memory of the smart devices. Figure
6 shows the normal CPU usage for Idle and Active statuses of the Raspberry Pi
device. The normal CPU usage for Idle devices is between $0.55$% to $0.88$%.
The memory usage of the Active smart devices is between $0.88$% to $1.50$%. At
the same time, the CPU usage is more than $1.5$% once we send the malicious
attack to the smart devices.

Figure 6: Raspberry Pi (CPU Usage Before and After the attack).

We also calculate the memory usage of another smart device (Arduino). The main
purpose of using two different devices is to show how the algorithm works for
different devices which implement different architectures. For printing the
memory usage of the Arduino device, we used a specific library to get the free
usage memory for different statuses of the smart device, e.g., Idle, Active,
under attack. Therefore, the memory usage for the first status, as shown in
Figure 7, fluctuates between $8.1$% to $11$%, and for the Active status, it is
between $11$% to less than $16$%. The memory usage percentage changes to more
than $17$% and less than $50$% once we send a malicious attack to the smart
device.

Figure 7: Arduino (Memory Usage Before and After the attack).

The results and analyzes of this experiment assisted us in understanding the
impact of the memory usage attack on smart devices and building a lightweight
algorithm to protect these devices from such an attack. The following section
describes the detection algorithm and presents some results.

##  6 Threat Mitigation

This paper introduces a detection mechanism and response to cyber-attacks on
smart devices’ memory usage. We also propose a lightweight algorithm to detect
such memory changes inside smart devices by monitoring memory usage. Once the
attack is detected, the algorithm will force the memory to stop listening to
such an attack (e.g., stop reading and writing to memory). We also disconnect
the victim devices from the Internet automatically. We implement this
algorithm in the smart devices themselves. The presented mechanism records the
response of the attack, and memory usage, for different states such as Idle,
Active, under attack. The detection algorithm detects any breach in the memory
usage of smart devices.

###  6.1 Proposed Algorithm

The attacker aims to consume more memory usage of the smart device, and the
monitoring mode of the presented algorithm updates and registers all different
cases of memory behaviors before and after the attack. We record the change on
memory for every $3$ second for $1$ minute. According to the data obtained
from the testbed, the attacker can change the memory usage within $67$% of
wrong values during $10$ minutes in total.

Algorithm 1 A Technique to detect Memory Usage Attack

1:Input: $d,Diff,C1,T1,Alert$

2:Output1: Normal($M1$)

3:Output2: Abnormal ($M2$)

4:Final Result: Output1 $or$ Output2

5:$M1:Reading_{memory-usage}$

6:$MEM(d)=f(mem(d),ATK,n)$

7:$M2:Reading_{memory-usage}$

8:$Reading_{Threshold}=Max_{usage(MEM)}-Min_{usage(MEM)}$

9:$Diff=M1-M2$

10:if $M2=M1$ then

11: if $Diff>Reading_{Threshold}$ then

12: $ResetT1$

13: if $Alert==^{\prime}On^{\prime}$ then

14: monitor memory

15: else

16: $C1=C1+1$

17: if $C1>Max_{(memory-usage)}$ then

18: $Alert==^{\prime}On^{\prime}$

19: Attack detected

20: Detect the main source ($X$)

21: Stopped Reading/Writing on Memory from $(X)$

22: Disconnect the smart device ($d$) from the Internet

23: else

24: Return back to monitor memory

25: else

26: if $C1>0$ then

27: $T1=T1+1$

28: if $T1>Threshold_{T}$ then

29: Reset Alert

30: Reset C1

31: Reset C1

32: Attack stopped

33: else

34: return back to monitor memory

This Algorithm 1 takes the recorded readings from the DB for each smart device
in the IoE system. The variable $Diff$ stores the subtraction of previous
(_before sending such attack_) and current (_after sending malicious attack_)
smart devices reading. For instance, the maximum memory usage of the smart
devices for Idle and Active smart devices are given in Figure 5. The variable
$Diff$ stores the subtraction of the previous and current memory usage
readings. For instance, the maximum sudden memory usage change expected in the
memory of the smart device is given by subtracting the value of the maximum
memory usage when the device is under attack minus the minimum memory usage
when the device is Active and Idle before sending any attack.

$Reading_{Threshold}=Max_{usage(MEM)}-Min_{usage(MEM)}$ (4)

When the variable $Diff$ exceeds the expected value, the variable $T1$
_"Timer"_ is reset, and we verify whether the alert message has been sent to
the administration. If not, we increase the $counter1$ variable, which records
the number of times the difference between previous and current memory usage
reading exceeds the maximum allowed value. When the $counter1$ variable is
greater than the maximum allowed value, it sends the alert message indicating
that the space of memory usage addressed to that smart device is under memory
usage attack. At this stage, we stop any reading/writing operations to and
from memory and disconnect the smart device from the Internet, as all victim
device’s IPs are stored in the black-list of our DB.

Through experimentation, we consider the scenario when the attacker stops the
attack. When the $Diff$ value is less than $Reading_{threshold}$ value, we
compare whether the variable counter is greater than $zero$, then we increase
$T1$. We can assume the attack stops if the variable is greater than the
$Time_{Threshold}$ variable. Finally, we reset the alert: $counter1$ and $T1$
variables.

After detecting the memory usage attack of such a device $(d)$, we put all the
victim devices on a black-list. Then, once the attack is detected on such a
device, we first stop any operation on the memory, e.g., read and write on
memory. We disconnect the Internet connection of the smart device $(d)$ to
prevent any further attack on the smart device’s memory usage. The next
section presents different results regarding detecting memory usage attacks.

Therefore, the mitigation is summarized in the following steps:

  1. 1.

add the victim smart devices’ IP to a black-list;

  2. 2.

stop any reading/writing to the smart device;

  3. 3.

disconnect the smart device from the Internet.

##  7 Experimentation and Discussion

###  7.1 Results

We ran malicious attacks on the smart device to check the memory usage before
and after the attack. Figure 8 shows the mechanisms of our algorithm to fetch
the attack once it is started. The monitoring mode of the memory usage sends
memory usage readings to the algorithm, and inside the algorithm, there is a
statistics comparison between normal and abnormal cases. As described in
Section 4, we first check the behavior of the smart devices once there is an
attack, and we register all different cases for memory usage, e.g., Idle,
Active, under attack. The main purpose of this analysis is to study the attack
first and then to build a mitigation mechanism to detect memory attacks.

Figure 8: Raspberry Pi (Detecting the memory usage Attack.)

Figure 8 and 9 shows the presented results of detecting the attack once it
starts; we can notice that the attack starts when the memory usage is greater
than $37$%, and the $Diff$ variable is greater than the expected memory usage
value. At this stage, the smart device $d$ is passed through different
operations, e.g., stop reading/writing on $d$, disconnect $d$ from the
Internet to stop any further attack, and send an alert to the administration
about the status of the smart device.

The detection algorithm also notified the administration once the attack
stopped. This stage will help with further operations.

Figure 9: Raspberry Pi (Detecting the memory usage once the attack starts and
when it stops).

Through this experiment, we also studied the behavior of the CPU usage of the
Raspberry Pi device under the same attack. Figure 10 shows the behavior of the
CPU usage before and after attacking the memory of the smart devices. We also
applied the same detection algorithm to study the behavior of the mitigation
algorithm on the CPU and whether this algorithm detects the attack or not.

Figure 10: CPU Usage during the attack when it started and stopped (Raspberry
Pi).

The same calculation is applied to the Arduino, and the detection algorithm
records different variables about the attack once it is started and stopped.
Figure 11 shows the recorded results of detecting the attack. We can notice
that the attack started when the memory usage percentage increased to be more
than $16$%, and for detecting the attack when it is stopped, once the memory
usage percentage decreased to be less than $20$%. Once the system detects that
the attack is stopped on the smart device $d$, the system might disconnect the
smart device from the Internet, or the actual attack is stopped from the main
source.

Figure 11: Arduino (Detecting the memory usage once the attack starts and when
it stops).

The algorithm also stores all victim devices’ IPs in the black-list, so when
there is an attack on the smart device, we disconnect the smart device to
prevent any further attack. We also prevent further access to the database
until the administration team solves the issue.

##  8 Conclusion and Future work

The Internet of Everything is the beginning of a new era of technology in
Internet-based smart communication and connecting smart devices. The security
of IoE pillars is important as some suffer from resource-constraints problems.
This paper proposed an approach that can detect and classify memory usage
attacks using memory-based features extracted from the memory usage of the
smart device. The approach represents a mitigation method to detect the attack
once it appears in the memory usage of the smart devices. First, we monitor
memory usage by using a specific tool in _Python_ script and _C_ language to
fetch different data about memory usage. Then, all the fetched data is stored
in the DB for further calculation. Second, we studied the attack behavior and
registered the memory usage readings before and after the attack. In this
work, we conduct static and dynamic analysis of the memory usage attack. In
particular, we have conducted all the experiments in an isolated and cost-
efficient experimental setup. It is observed that malicious attacks, e.g.,
flooding attacks have a significant impact on the resources of the IoE smart
devices. When an IoE edge device is flooded with malicious attacks, there are
significant increases in CPU and memory usage. This analysis helps in building
the detection algorithm. The detection method relies on monitoring the memory
usage to compare different variables of the memory reading. It is also able to
detect the attack on time once it happens. Moreover, it can detect if the
intruder stops the attack or not. We also build an alert message inside the
algorithm to send different notifications to the administration once the
attack is detected. Moreover, all victim devices are disconnected from the
Internet, and all read/write operations to and from memory are also stopped.
In the future, we will focus on detecting the main sources of memory usage
attacks in the IoE environment.

## References

  * [1] Aghaeikheirabady, M., Farshchi, S.M.R., Shirazi, H.: A new approach to malware detection by comparative analysis of data structures in a memory image. In: 2014 International Congress on Technology, Communication and Knowledge (ICTCK). pp. 1–4. IEEE (2014)
  * [2] attack, D.: Ddos attack that disrupted internet was largest of its kind in history, experts say (2016), https://www.theguardian.com/technology/2016/oct/ 26/ddos-attack-dyn-mirai-botnet
  * [3] Dai, Y., Li, H., Qian, Y., Lu, X.: A malware classification method based on memory dump grayscale image. Digital Investigation 27, 30–37 (2018)
  * [4] Duan, Y., Fu, X., Luo, B., Wang, Z., Shi, J., Du, X.: Detective: Automatically identify and analyze malware processes in forensic scenarios via dlls. In: 2015 IEEE International Conference on Communications (ICC). pp. 5691–5696. IEEE (2015)
  * [5] Jamil, B., Ijaz, H., Shojafar, M., Munir, K., Buyya, R.: Resource allocation and task scheduling in fog computing and internet of everything environments: A taxonomy, review, and future directions. ACM Computing Surveys (CSUR) (2022)
  * [6] Jara, A.J., Ladid, L., Skarmeta, A.: The internet of everything through ipv6: an analysis of challenges, solutions and opportunities. Journal of Wireless Mobile Networks, Ubiquitous Computing, and Dependable Applications 4, 97–118 (09 2013)
  * [7] Kathole, A.B., Katti, J., Dhabliya, D., Deshpande, V., Rajawat, A.S., Goyal, S., Raboaca, M.S., Mihaltan, T.C., Verma, C., Suciu, G.: Energy-aware uav based on blockchain model using ioe application in 6g network-driven cybertwin. Energies 15(21), 8304 (2022)
  * [8] Mosli, R., Li, R., Yuan, B., Pan, Y.: Automated malware detection using artifacts in forensic memory images. In: 2016 IEEE Symposium on Technologies for Homeland Security (HST). pp. 1–6. IEEE (2016)
  * [9] Mosli, R., Li, R., Yuan, B., Pan, Y.: A behavior-based approach for malware detection. In: IFIP International Conference on Digital Forensics. pp. 187–201. Springer (2017)
  * [10] Murtuza, S.: Internet of everything: Application and various challenges analysis a survey. In: 2022 1st International Conference on Informatics (ICI). pp. 250–252 (2022). https://doi.org/10.1109/ICI53355.2022.9786891
  * [11] Rani, R., Kumar, S., Kaiwartya, O., Khasawneh, A.M., Lloret, J., Al-Khasawneh, M.A., Mahmoud, M., Alarood, A.A.: Towards green computing oriented security: A lightweight postquantum signature for ioe. Sensors 21(5), 1883 (2021)
  * [12] Rathnayaka, C., Jamdagni, A.: An efficient approach for advanced malware analysis using memory forensic technique. In: 2017 IEEE Trustcom/BigDataSE/ICESS. pp. 1145–1150. IEEE (2017)
  * [13] Ryoo, J., Kim, S., Cho, J., Kim, H., Tjoa, S., Derobertis, C.: Ioe security threats and you. In: 2017 International Conference on Software Security and Assurance (ICSSA). pp. 13–19 (2017). https://doi.org/10.1109/ICSSA.2017.28
  * [14] Shi, W., Xu, W., You, X., Zhao, C., Wei, K.: Intelligent reflection enabling technologies for integrated and green internet-of-everything beyond 5g: Communication, sensing, and security. IEEE Wireless Communications (2022)
  * [15] Sihwail, R., Omar, K., Zainol Ariffin, K.A., Al Afghani, S.: Malware detection approach based on artifacts in memory image and dynamic analysis. Applied Sciences 9(18), 3680 (2019)
  * [16] Vomel, S., Freiling, F.C.: A survey of main memory acquisition and analysis techniques for the windows operating system. Digital Investigation 8(1), 3–22 (2011)
  * [17] Wei, L., Wu, J., Long, C., Lin, Y.B.: The convergence of ioe and blockchain: Security challenges. IT Professional 21(5), 26–32 (2019)
  * [18] Zaki, A., Humphrey, B.: Unveiling the kernel: Rootkit discovery using selective automated kernel memory differencing. Virus Bulletin pp. 239–256 (2014)
  * [19] Zhan, J., Dong, S., Hu, W.: Ioe-supported smart logistics network communication with optimization and security. Sustainable Energy Technologies and Assessments 52, 102052 (2022)
  * [20] Zhang, H., Shlezinger, N., Guidi, F., Dardari, D., Imani, M.F., Eldar, Y.C.: Near-field wireless power transfer for 6g internet of everything mobile networks: Opportunities and challenges. IEEE Communications Magazine 60(3), 12–18 (2022)


# Deploying machine learning to assist digital humanitarians: making image
annotation in OpenStreetMap more efficient

John E. Vargas-Muñoza*, Devis Tuiab, Alexandre X. Falcãoa *First author.
Email<EMAIL_ADDRESS>a Laboratory of Image Data Science, Institute
of Computing,
University of Campinas, Campinas, Brazil;
b Laboratory of Geo-information Science and Remote Sensing,
Wageningen University & Research, the Netherlands

###### Abstract

This is a preprint of the paper published in the International Journal of
Geographical Information Science. The publisher version can be found here:
https://doi.org/10.1080/13658816.2020.1814303
Locating populations in rural areas of developing countries has attracted the
attention of humanitarian mapping projects since it is important to plan
actions that affect vulnerable areas. Recent efforts have tackled this problem
as the detection of buildings in aerial images. However, the quality and the
amount of rural building annotated data in open mapping services like
OpenStreetMap (OSM) is not sufficient for training accurate models for such
detection. Although these methods have the potential of aiding in the update
of rural building information, they are not accurate enough to automatically
update the rural building maps. In this paper, we explore a human-computer
interaction approach and propose an interactive method to support and optimize
the work of volunteers in OSM. The user is asked to verify/correct the
annotation of selected tiles during several iterations and therefore improving
the model with the new annotated data. The experimental results, with
simulated and real user annotation corrections, show that the proposed method
greatly reduces the amount of data that the volunteers of OSM need to
verify/correct. The proposed methodology could benefit humanitarian mapping
projects, not only by making more efficient the process of annotation but also
by improving the engagement of volunteers.

###### keywords:

Interactive annotation, Very high resolution mapping; convolutional neural
networks; OpenStreetMap; volunteered geographical information; vector maps
update.

††articletype: Journal

##  1 Introduction

A large amount of mapping information of buildings has been collected in open
and commercial mapping services like OpenStreetMap and Google maps. However,
the building mapping data is concentrated mainly in urban areas. Recently,
humanitarian organizations like, the Humanitarian OSM Team (HOT) and the Red
Cross have created projects to map buildings in rural areas of developing
countries, in order that Non-Governmental Organizations (NGOs) can use the
maps to plan actions in response to crises that affect those areas. For
instance, the HOT Task manager project 111https://tasks.hotosm.org/ provides a
platform that includes a web-based map editor in which volunteers manually
annotate buildings and other classes of interest in some predefined areas in
OSM. The volunteers verify/annotate all the tiles in a selected geographical
area by observing aerial imagery. This is a time-consuming task since most of
the tiles do not contain the objects of interest in rural areas.

In general, geographical objects in OSM are defined by their geometries (e.g.,
polygon) and associated tags (e.g., building). In this work, we refer to OSM
geographical objects as OSM _annotations_. Figure 1 shows rural building
annotations in OSM that were manually digitized by volunteers. Although some
annotations are of good quality, several others have issues of misalignment or
incompleteness with respect to aerial imagery. Misalignment problems are
present because of image co-registration issues. The incompleteness is due to
new buildings that become visible in the updated imagery and also due to
incomplete annotations that might have been submitted by volunteers.
Additionally to these issues, modifications of the building can also result in
geometry changes.

|  |
---|---|---
(a) | (b) | (c)
|  |
(d) | (e) | (f)
Figure 1:  Examples of rural building annotations (orange circles), performed
by volunteers in OpenStreetMap, superimposed over Bing imagery. a-c) correct
annotation, d) annotations with misalignment errors, e) annotations that do
not correspond to any building in the aerial images, f) incomplete
annotations.

Deep learning methods are nowadays considered state-of-the-art techniques for
several computer vision tasks, such as object detection and segmentation (He
_et al._ 2017, Liu _et al._ 2018). These techniques have been also used
successfully for building segmentation in aerial images (Maggiori _et al._
2017, Saito _et al._ 2016, Volpi and Tuia 2017, Cheng _et al._ 2019) and have
led to accurate results in urban areas (Hamaguchi and Hikosaka 2018, Wang _et
al._ 2017). These methods could be useful for automatic building extraction in
large settlements222https://github.com/Microsoft/USBuildingFootprints.
However, the main drawback of deep learning methods is the requirement of a
large amount of labeled data for training, usually in the range of several
thousands of labeled samples. OSM data have been used for several applications
including real-time routing (Luxen and Vetter 2011), autonomous driving
(Fleischmann _et al._ 2017), 3D building modeling (Wang and Zipf 2017) and
landuse classification (Srivastava _et al._ 2018, 2019). Audebert _et al._
(2017) and Kaiser _et al._ (2017) used aerial images and OSM footprint
annotations to train deep learning-based methods to perform semantic
segmentation of buildings and road networks. More recently, Chen _et al._
(2018) proposed a deep learning-based building detection method that used data
from three open crowdsourced geographic systems: OSM, MapSwipe
333https://mapswipe.org/, and OsmAnd 444https://osmand.net/. However, the two
last resources just provided annotations at the tile level, useful for
building patch detection but not for building segmentation. Active learning
methods were proposed in Chen _et al._ (2018) and Chen and Zipf (2017) to
select better image patches for annotation and then train an effective
classifier from those patches. However, these works proposed solutions for
patch-based building detection which cannot be directly used to automatically
update maps of open mapping services, like OSM and Wikimapia
555http://wikimapia.org, because those maps store vectorial footprints that
delineate individual buildings.

Although OSM data has been used to train machine learning methods for some
applications, the OSM annotations are very often of not good quality (Mnih and
Hinton 2012, Jilani _et al._ 2014, Ali _et al._ 2014, Vargas-Muñoz _et al._
2019). For instance, Mnih and Hinton (2012) found noisy labels (e.g.,
incomplete labels) of road networks and buildings that were used to create
semantic segmentation methods. Jilani _et al._ (2014) observed that there
exist errors in the tags associated with different types of road networks in
OSM data, and proposed a method for automatic road tagging. In other works,
Ali _et al._ (2014, 2017) found several green areas of different classes
(e.g., meadows, grass, parks, and gardens) that have incorrect labels in OSM.
In the specific case of building annotations, Vargas-Muñoz _et al._ (2019)
identified three main issues in building annotations of OSM: i) location
inaccuracies (see an example in Figure 1d), ii) annotations not related to a
building (see Figure 1e) and iii) missing annotations (see Figure 1f).

In addition to the problem of low data quality in OSM, the amount of available
labeled data is small in rural areas as compared to urban regions. This
happens because the number of volunteers that update OSM data drops outside
cities (Neis and Zielstra 2014): usually these areas get attention of the
volunteers only when an epidemy or a natural disaster happens there. This
makes more difficult to train accurate models for the segmentation of
buildings in rural areas. Vargas-Muñoz _et al._ (2019) proposed a methodology
to automatically correct rural building annotations with the aforementioned
problems in OSM. However, the accuracies attained by this method are not
sufficient for production-ready automatic updates of OSM.

An additional problem for the methods that try to automate the process of
updating automatically OSM building data is that the output of such methods
should be vectorial building footprints. Recently, Marcos _et al._ (2018)
proposed a method that outputs vectorial building delineations using a method
based on Active Contour Models (ACM), which uses a Convolutional Neural
Network (CNN) to learn the parameters of the ACM model. A different approach
is proposed in Tasar _et al._ (2018) via a mesh-based approximation method
that converts binary building classification maps into polygonized buildings.
Vargas-Muñoz _et al._ (2019) proposed a CNN that detects rural buildings of
predefined shapes, which can be easily exported to vectorial format. All these
methods strongly depend on the accuracy of the previously computed building
map.

In this work, we propose a methodology for interactive correction of rural
buildings in OSM. Our proposed method starts by correcting misalignment errors
of the annotations using a Markov Random Field (MRF) based method proposed in
Vargas-Muñoz _et al._ (2019). We selected that approach, because it
outperforms other methods for the particular task of building footprint
alignment. From that moment on, the model interacts with a human operator for
digitizing objects in OSM. The model selects the image locations that maximize
the chances of editing corrections.

In this way, the user just analyzes a small set of selected regions of a large
geographical area. Our aim is to reduce the effort of the user for mapping
buildings in a certain geographical area, by intelligently selecting just the
regions that require user corrections/annotations. The selection of these
regions is determined by analyzing a building probability map obtained by a
CNN method on aerial images and the current OSM annotations. Given that large
areas need to be analyzed to select the most interesting regions for
annotation, we propose an efficient CNN model for building segmentation that
performs fast inference. To do so, we enrich our CNN with a branch for early
stopping when there are no buildings in the analyzed image patch. Whenever a
considerable amount of new labeled data is available, the CNN model is
retrained to improve the accuracy of the building predictor, and so the
accuracy of the selection of tiles that need annotations/corrections.
Experiments performed by simulation and real human annotations in datasets of
different countries (four different datasets) show that the proposed approach
effectively reduces the number of tiles that need to be screened and therefore
the human workload. In most of our experiments, we found that our method
allows to correct 98% of the wrong annotations by analyzing less than a
quarter of the data.

In Section 2, we present the proposed methodology for interactive
annotation/correction of OSM rural building annotations. Section 3 presents
the data set and the setup used in our experiments, and Section 4 shows the
experimental results. Finally, Section 5 presents the conclusions of the
paper.

##  2 Methodology

Our proposed methodology is illustrated in Figure 2. First, we train a CNN
building segmentation model to obtain a building probability map, which is
used to perform the alignment of the original OSM polygons. Then, the image
being analyzed is split into tiles, in order to select a small number of
regions that need manual editing in OSM. This is done based on the probability
map and the currently aligned polygons. After the annotation of the selected
samples, a stopping criterion is verified to finish the process of annotation.
If this criterion is not met, the new building annotations are used to improve
the building classification CNN, which in turn improves the process of
selection of tiles in the next iteration. The next sections explain in detail
the steps of the proposed methodology.

Figure 2: The proposed methodology for interactive correct OSM rural building
annotations. The orange circles represent misaligned OSM building annotations
while the blue circles represent the annotations after the misalignment
correction.

###  2.1 Computation of the building probability map

Given that our model is applied to large areas where rural buildings are
sparsely located, we propose an efficient CNN-based method for building
segmentation. Our proposed model is based on the U-Net architecture proposed
in Ronneberger _et al._ (2015). U-Net has shown good performance in several
applications and recently an ensemble of U-Net models has obtained the best
performance in a building segmentation competition, the DeepGlobe challenge
Demir _et al._ (2018). The U-Net model (illustrated in the gray boxes of
Figure 3) consists of several operations that extract image features and
capture contextual information, alongside a symmetric set of operations that
upsample the feature maps, therefore enabling precise pixel-level semantic
labeling. We extend the U-Net model by adding an extra branch after the third
set of convolution groups (see Figure 3a). This branch of the model considers
that the image features extracted until that point are sufficient to identify
whether the image contains buildings. A fully connected layer, denoted as
$FC$, is applied over the aforementioned features to perform binary
classification (i.e., there are buildings in the patch or not). In order to
train the model with the additional branch, we define a new loss function
illustrated in Figure 3b. Our loss function is defined as the sum of a
detection and segmentation loss, both computed using binary cross entropy:

$\displaystyle L_{seg}(\hat{y}^{s},y^{s})$ $\displaystyle=$
$\displaystyle-\frac{1}{N}\sum_{i}^{N}y^{s}_{i}\log(\sigma(\hat{y}^{s}_{i}))+(1-y^{s}_{i})\log(1-\sigma(\hat{y}^{s}_{i}))$
(1) $\displaystyle L_{det}(\hat{y}^{d},y^{d})$ $\displaystyle=$
$\displaystyle-\frac{1}{M}\sum_{j}^{M}y^{d}_{j}\log(\sigma(\hat{y}^{d}_{j}))+(1-y^{d}_{j})\log(1-\sigma(\hat{y}^{d}_{j}))$
(2) $\displaystyle L(\hat{y}^{s},\hat{y}^{d},y^{s},y^{d})$ $\displaystyle=$
$\displaystyle L_{seg}(\hat{y}^{s},y^{s})+L_{det}(\hat{y}^{d},y^{d}),$ (3)

where $\sigma$ is the sigmoid function, $\hat{y}^{s}$ is the output of the
last convolutional layer of the U-Net model and $y^{s}$ is the segmentation
ground truth map, $\hat{y}^{d}$ is the detection prediction obtained from $FC$
and $y^{d}$ is the detection ground truth, $N$ is the number of samples in the
processed batch of pixels (segmentation output) and $M$ is the number of
samples in the processed batch of images.

---
(a) Loss computation
(b) Proposed strategy for fast inference
Figure 3: Proposed method for building segmentation: a) Loss computation based
on a detection and segmentation loss b) Fast inference by using a early
exiting brach.

During inference, the proposed model computes sequentially the first three
groups of convolutions followed by a fully connected layer and a sigmoid
function that outputs the building presence probability value. If this value
is higher than a certain threshold $\theta$, then we consider that there exist
buildings in the analyzed patch and the rest of the U-Net model is executed to
obtain a building segmentation map. Otherwise, we consider that there are no
buildings in the image patch and the model outputs an empty probability map.
In this way, the inference time is reduced.

###  2.2 Correction of misalignment errors

In order to correct alignment errors in OSM, we use the approach proposed in
Vargas-Muñoz _et al._ (2019) that showed good performances for the alignment
of rural building annotations. In that work, the OSM polygons of rural
buildings are grouped based on spatial proximity. Then, groups of buildings
are aligned with a single shift vector. Aligning groups of polygons makes the
method more robust to building probability maps of bad quality. The
correlation of the aligned annotations and building probability maps is used
to measure the performance of a given alignment vector. Given that nearby
groups of buildings have similar registration errors, Vargas-Muñoz _et al._
(2019) used MRF to find the alignment vectors
$\mathbf{d}=\\{d_{0},d_{1},\ldots,d_{n}\\}$ that can correct the current
annotation locations $\mathbf{x}$ using a building probability map
$\mathbf{y}$. The optimum alignment vectors minimize the following energy
function:

$\displaystyle\mathbf{\hat{d}}=\arg\min\limits_{\mathbf{d\in\mathbf{v}^{\mathcal{N}}}}\sum\limits_{i}-\log
C(A(x_{i},d_{i}),y_{i})+\beta\sum_{j\in\mathbf{h}_{i}}\frac{1}{\mathcal{Z}}||d_{i}-d_{j}||_{2},$
(4)

where the unary term, that measures the matching between the annotations and
probability map, is computed by using the normalized dot product
$C(A(x_{i},d_{i}),y_{i})$ between the aligned annotations $A(x_{i},d_{i})$
($A$ is a function that shifts the position of the annotation $x_{i}$ using
the alignment vector $d_{i}$) and the probability map $y_{i}$, while the
pairwise term is obtained by computing the vector norm of the difference of
the alignment vector $d_{i}$ and the neighboring alignment vectors $d_{j}$
($j\in\mathbf{h}_{i}$). In equation 4
$\mathbf{v}=\\{v_{1},v_{2},\ldots,v_{p}\\}$ represents the set of all the
alignment vectors, $\mathcal{Z}$ is a normalization term computed as the
maximum possible distance between two alignment vectors in $\mathbf{v}$ and
$\beta$ is a regularization parameter. We use the Iterative Conditional Modes
(ICM) algorithm to find the solution of equation 4.

###  2.3 Selection of tiles for correction/verification

Once the building probability map has been generated and the existing OSM
annotations have been aligned to it, volunteers can start the editing process
in OSM. The aim of this section is to define a criterion to focus the
volunteers’ efforts where it really matters (regions with wrong or missing
annotations). In this respect, it is crucial to obtain a measure of how
incorrect an annotation is, based on the current OSM annotations and a
building probability map. Such measure is used to create an ordered list of
tiles $L$, where those whose annotations have more chances to be incorrect are
ranked higher. Then, the top $t$ tiles from $L$ are selected for
correction/verification by a user. We convert the OSM vectorial polygons to
images with value one in the positions that are inside a building annotation,
otherwise, zero everywhere else. This is done in order to be able to compare
the OSM annotations with the building probability map. In the next sections,
we describe the measures proposed.

Mutual information

Mutual information (MI) has been used as an effective similarity metric to
compare images Pluim _et al._ (2003). The MI of two variables measures the
amount of information that one variable carries about the other. The MI of two
images $A$ and $B$ can be defined as follows:

$MI(A;B)=\sum_{b\in B}\sum_{a\in
A}p(a,b)\log{\left(\frac{p(a,b)}{p(a)\,p(b)}\right),}$ (5)

where $p(a,b)$ is the joint probability distribution of the pixels that
correspond to $A$ and $B$ and $p(a)$ and $p(b)$ are the marginal probability
distributions of $A$ and $B$, respectively. For the MI metric, the lower the
value the higher the priority to be selected for verification/correction.

Normalized dot product

Vargas-Muñoz _et al._ (2019) used the normalized dot product (NDP) to measure
the degree of matching between the building annotations and the building
probability map. This measure is defined as follows:

$NDP(A;B)=\frac{A\cdot B}{size(A)},$ (6)

where $A$ and $B$ are images of the same size, ($\cdot$) represents the dot
product operation and $size(A)$ is the number of pixels in image $A$. For the
NDP metric, the lower the value the higher the priority to be selected for
verification/correction.

Sum of absolute differences

The sum of absolute differences (SAD) of the pixel values of two images is an
efficient and effective way to measure the degree of matching of two images,
as shown in Alsaade (2012). For the SAD metric, the higher the value the
higher the priority to be selected for verification/correction.

###  2.4 User annotation and evaluation of the stopping criteria

The user verifies/corrects the annotations in the selected tiles. In order to
correct annotations, the user applies three types of manual operations that
are available in a Graphical User Interface (GUI), for example, the web-based
iD editor 666http://ideditor.com/ of OSM: align (drag and drop annotations
into the right position), remove and add (digitize manually new polygons). The
user performs these operations by visually inspecting the original OSM
annotations superimposed over the aerial imagery.

After the user verification/correction of the selected tiles, a stopping
criterion is evaluated. We propose as criterion the percentage of tiles that
required correction since the last $k$ analyzed tiles, denoted as $p_{k}$.
This measure is updated after every tile is analyzed by the user. Note that
the tiles are selected from a list $L$, sorted in ascending order in terms of
expected correctness of the annotations available, estimated with measures
presented in Section 2.3. Thus, the more tiles are analyzed by the user, the
lower the value of $p_{k}$. This happens because, eventually, the majority of
the tiles with incorrect annotations will be already analyzed and the majority
of the remaining tiles will not need any correction. Once $p_{k}$ is lower
than a given threshold $r_{k}$ the process of user verification/correction
ends.

After a considerable amount of tiles (e.g., 20) are verified/annotated by the
user, the proposed CNN building segmentation model is fine-tuned with the new
annotated data. This updated model is then used to improve the building
probability maps, which in turn improves the performance of the annotation
correctness measures described in Section 2.3.

##  3 Data and experimental setup

###  3.1 Datasets

For the validation of our proposed methodology, we used data obtained from the
countries of the United Republic of Tanzania and the Republic of Zimbabwe. The
CNN building segmentation model was trained with $3134$ OSM rural buildings
annotations from several regions of Tanzania. These OSM annotations for these
regions were manually verified/corrected on a set of Bing aerial images, that
cover $23.75$ km2, with $30$cm spatial resolution, acquired over the Geita,
Singida, Mara, Mtwara, and Manyara regions of Tanzania.

To evaluate our proposed methodology with simulations of user annotation
corrections we collected two test datasets, spatially disjoint with respect to
the training set. The first dataset is collected in the country of Tanzania
and is composed of $1267$ OSM building footprints (in an area of 25.5 km2)
that contain annotation errors. These annotations were located close to the
region of Mugumu in Tanzania, this data is called the Tanzania dataset in
Section 4. The second dataset contains $1392$ OSM annotations (in an area of
27.9 km2), that also contains errors, is collected in the region of Midlands
in Zimbabwe, and is called Zimbabwe dataset in Section 4. In order to perform
simulations of user annotations, we manually corrected the annotations in
these two datasets.

In order to perform experiments with real user interactions, we used OSM
annotations from two regions. The first one located close to Mugumu region in
Tanzania, denominated as Mugumu dataset (646 building footprints collected
from an area of 9.4 km2). The second evaluation area is located in the Gweru
region, in Zimbabwe, denominated as Gweru dataset (1273 building footprints
collected from an area of 28.4 km2).

###  3.2 Model setup

The CNN model was trained for $20$ epochs with an initial learning rate of
$0.001$ decreased by a factor of $0.1$ after every $10$ epochs. During the
interactive annotation process, we finetuned the CNN model for $10$ additional
epochs and a learning rate of $0.0001$ with the new annotated data. For our
proposed CNN model, we used $\theta=0.1$ to predict if an image patch contains
or not buildings. As in Vargas-Muñoz _et al._ (2019), we used $\beta=2.0$ for
the alingment correction process. We split the images corresponding to the
selected geographical regions in tiles of size $256\times 256$ pixels. This
tile size was chosen because we observed that it is large enough to cover
groups of rural buildings with little background information. In order to
evaluate the proposed stopping criteria, we fixed the parameter value $k=100$
and evaluated the performance of different values
$r_{k}\in\\{0.02,0.05,0.10\\}$. For the experiments performed to choose the
tile selection strategy, we took into consideration a common tricky case that
affects the metrics MI and NDP. In the case where there are no annotations in
a tile but there are a small amount of false positive predictions, the
aforementioned metrics will output zero. Then, the method will assign a high
priority to the tiles to be verified/corrected by a user, which is not correct
since there is nothing to edit in that tile. Thus, for MI and NDP, we verified
if the tile does not contain OSM annotations, and if the CNN prediction has a
building with size shorter than a very small building in our dataset (20
pixels of size). If that is the case we output the value one which will assign
a low selection priority to the tile.

###  3.3 Setup for experiments with real user annotation corrections

So far, the proposed methodology has considered simulated annotation
corrections to obtain perfect annotations. In order to evaluate our proposed
methodology in a realistic scenario, we also performed a set of experiments
with real volunteers performing annotations. To run those tests we deloped a
web application that allows several users to correct annotations in
OpenStreetMap, by using the iD editor 777http://ideditor.com/ API. Given a
geographical area where we want to correct the rural building annotations, our
method selects the tiles that need to be verified/corrected by the user with
higher priority. When a user asks for a tile for analysis, the web application
loads a tile highlighted with a bounding box in magenta in the iD editor.
Figure 4a depicts the bounding box of the selected tile superimposed over
aerial imagery and the current rural building annotation at that location. The
user is asked to correct/verify existing annotations, and also to add new
building annotations if needed. For instance Figure 4b shows the annotations
after user corrections, where two missing buildings were added on the bottom
left of the bounding box. For these experiments, 9 volunteers used the web
application to interactively correct OSM rural building annotations, in March
2019. The volunteers were students of the Computer Science department, at the
University of Campinas, Brazil. They were unfamiliar with OSM and the digital
annotation tasks. After an introduction session to the editing environment,
they were asked to perform around 7 hours of annotations in two working
sessions (labeling the Mugumu and Gweru dataset).

It is difficult to ensure that the new annotations are of good quality. This
depends a lot on the volunteers’ skills. Thus, it was very important to
perform an introductory tutorial of the building editing tool (i.e., the iD
editor of OSM). Such quality checks can be implemented in real deployments of
the system (e.g. using golden questions or evaluating users’ behaviour (e.g.
Gomes _et al._ (2011)), but this goes beyond the scope of this paper and will
be considered in future research.

|
---|---
(a) | (b)
Figure 4: Gaphical User Interface of the iD editor for editing OpenStreetMap
annotations: a) Tile (deliminted by the bounding box in magenta) loaded for
verification/correction of rural building annotations, b) OSM map after the
annotation of two new buildings in the bottom left part of the tile.

##  4 Results

In this section, we present the results obtained by the proposed methodology.
Section 4.1 compare the proposed CNN model with the standard U-Net model.
Using the computed probability maps obtained by our proposed CNN method we
correct alignment errors, as explained in Section 2.2. Section 4.2 evaluates
the alignment results. Then, after correcting existing alignment errors in
Section 4.3 we compare several approaches to select the tiles that require
correction. Section 4.4 evaluates the proposed stopping criteria by analyzing
the percentage of wrong annotations corrected and the percentage of analyzed
tiles when the stopping criteria is met. In Section 4.5, we report the results
of the experiments including real annotation corrections.

###  4.1 Evaluation of the proposed CNN method to compute a building
probability map

We compared the efficiency and efficacy of the proposed CNN method with the
standard U-Net model, in the Tanzania dataset. The computed probability maps
were thresholded to obtain the pixel-based binary classification map (i.e.,
greater than 0.5 building pixel otherwise background pixel). For the
comparison, we used precision, recall, and F1-score at the object-level, as in
Vargas-Muñoz _et al._ (2019). Both models were trained with the configuration
explained in Section 3.2. Table 1 shows that the proposed method attains
better performance than U-Net, in the three metrics. It is also more than 70%
faster than the U-Net model, reducing the processing time from 157 to 90 secs
in the evaluated dataset. Our proposed method performs first detection
(classifying if an image patch contains buildings or not) and performs
semantic segmentation (pixel-based classification) just if needed. It might
happen that some image patches classified as not containing buildings actually
contain buildings and semantic segmentation should be applied to them. Thus,
we measure the detection recall to verify that our model is not having many
errors of this type. Table 1 shows the detection recall of our proposed CNN
method, which attains a very high value.

Table 1: Performance of the proposed CNN method for building detection and segmentation as compared to the standard U-Net model.  Methods | Tanzania dataset
---|---
| Precision | Recall | F1-score | Time(sec) | Detection recall
Proposed | 0.47 | 0.84 | 0.60 | 90.0 | 0.99
U-Net | 0.43 | 0.72 | 0.54 | 157.0 | -

###  4.2 Evaluation of the alignment method

We applied the alignment method, presented in Section 2.2, to the original OSM
annotations of the Tanzania dataset. In order to measure how well this method
performs we computed the object-level accuracy of overlapping annotations with
the ground-truth. This is computed as the number of annotations that have a
strong overlap with a ground-truth annotation, Intersection over Union (IoU)
greater than 0.5 as in Vargas-Muñoz _et al._ (2019), divided by the number of
annotations that have at least a very small overlap with a ground-truth
annotation (IoU greater than 0.05). The object-level overlapping accuracy of
the aligned annotations is 93.9%, which is much better than 12.5% obtained by
the original OSM annotations (without alignment). Figure 5 presents visual
examples of OSM annotations before and after alignment. The first two rows in
Figure 5 show examples that by aligning the annotations we obtain a correct
set of annotations in those locations. The third row shows an example where
there is just one OSM annotation when there are many buildings in the aerial
imagery. Thus, after applying alignment there are still buildings that need to
be digitized by annotators. The fourth row presents an example where all the
buildings seem to have a circular shape but the leftmost annotation have a
rectangular shape. Thus, after performing alignment the annotator will still
need to remove the polygon with wrong shape and digitize a new building
annotation.

Example 1
---
|  |
(a) Image | (b) Original annotations | (c) Aligned annotations
Example 2
|  |
(d) Image | (e) Original annotations | (f) Aligned annotations
Example 3
|  |
(g) Image | (h) Original annotations | (i) Aligned annotations
Example 4
|  |
(j) Image | (k) Original annotations | (l) Aligned annotations
Figure 5: Examples of alignment results in the Tanzania dataset. The original
misaligned annotations are presented in orange and the aligned annotations in
blue.

###  4.3 Tile selection strategy

After correcting the alignment of the original OSM annotations, we evaluated
the different selection strategies presented in Section 2.3, to find which
approach can better rank the tiles that require corrections, and therefore
minimize the effort of the user to verify/correct rural building annotations
in OSM. We included as a baseline a method that randomly selects tiles to be
annotated by the user and also show an upper bound that uses the ground-truth
(GT) data to select the tiles. Figure 6 shows the results of our proposed
methodology in the Tanzania dataset using different strategies to select tiles
for annotation: MI, NDP and SAD. We also show a strategy, called Active SAD,
that uses the SAD to select tiles and then updates the CNN segmentation model
with the new verified/annotated data (i.e., the method using the complete
interactive pipeline in Figure 2). We can observe that SAD performs
considerably better than MI and NDP and Active SAD performs better than SAD in
the last iterations when a great part of the tiles with errors have been
discovered and corrected. Figure 7 shows the results of the evaluated methods
in the Zimbabwe dataset. The same trend can be observed in this dataset, SAD
performs better than the other selection methods. However, in this dataset
Active SAD is considerably better than SAD. This happens because the building
probability maps of the Zimbabwe dataset are less accurate than the
probability maps obtained for the Tanzania dataset. Therefore updating the
building segmentation CNN model has more impact on the improvement of the
selection strategy. Retraining the model with the new corrected annotations
adapts the model to the dataset and reduces domain adaptation problems Matasci
_et al._ (2012).

Figure 6: Results on the Tanzania dataset of our proposed methodology for
interactive annotation of rural buildings in OSM using several measures for
error annotation detection. The vertical axis of the plot shows the number of
annotation errors detected by the evaluated strategy and the horizontal axis
represents the number of tiles analyzed by the user. Figure 7: Results on the
Zimbabwe dataset of our proposed methodology for interactive annotation of
rural buildings in OSM using several measures for error annotation detection.
The vertical axis of the plot shows the number of annotation errors detected
by the evaluated strategy and the horizontal axis represents the number of
tiles analyzed by the user.

###  4.4 Stopping criteria

We evaluated the stopping criteria in the Tanzania and Zimbabwe datasets. For
the experiments, we used the method Active SAD, since it showed the best
performance among the other analyzed tile selection approaches (see Section
4.3). Table 2 shows the percentage of wrong annotations corrected and the
percentage of tiles analyzed when the stopping criterion is met. Note that %
corrected is computed as the number of wrong annotations shown for user
correction divided by the total number of wrong annotations. We can observe
that in the Tanzania dataset, when the annotation process stops, most of the
wrong annotations were already corrected and around 15% of the tiles were
analyzed for all the three values of $r_{k}$. In the Zimbabwe dataset
$r_{k}=10\%$ stops too early the annotation process, and leads to the
correction of just around half of the wrong annotations. Using $r_{k}=2\%$,
almost all the wrong annotations are detected but $20.9\%$ of the tiles are
analyzed. The difference in the results in the two datasets is mainly due to
the difference in the accuracy of the respective building probability maps.

Table 2: Percentage of wrong annotations corrected (% corrected) and percentage of tiles analyzed for three different values of $r_{k}$ (stopping criteria parameter) in the Tanzania and Zimbabwe datasets. $r_{k}$ (%) | Tanzania dataset | Zimbabwe dataset
---|---|---
| % corrected | % tiles analyzed | % corrected | % tiles analyzed
10 | 99.7 | 14.8 | 52.8 | 6.0
5 | 99.7 | 14.9 | 87.6 | 16.3
2 | 99.7 | 15.0 | 98.4 | 20.9

We also evaluated the behavior of the stopping criteria in datasets that have
annotations of different quality levels. To do that we validated the
performance of the proposed heuristic in several datasets obtained by
simulations of removal, addition, and random small shifts of buildings
annotations. For a given initial set of annotations, we randomly add and
remove a certain number of annotations and then randomly shift the annotations
in the horizontal and vertical axis in the range of 0 to 2 pixels. We
performed these operations over the Tanzania and Zimbabwe datasets by adding
and removing a percentage of the initial number of annotations.

Figure 8 shows the results of our proposed method with different stopping
criteria parameter values on the Tanzania and Zimbabwe datasets with different
quality levels of simulated wrong annotations (different percentages of
simulated additions and removals). In the Tanzania dataset (see Figures
8a-b)), we can observe that even in the case of very low quality annotations
our method can find almost all the wrong annotations for the three differents
$r_{k}$ values (stopping criteria). This at the cost of analyzing around
14-17% of the tiles in the Tanzania dataset.

In the Zimbabwe dataset (see Figures 8c-d)), we can observe that the
percentage of wrong annotations corrected varies with the quality level of the
annotations and the $r_{k}$ values. When the quality of the annotations is
reasonably good (10% of simulated additions and removals) the annotation
process stops too early because the probability maps are not of good quality
in this dataset and it is more difficult to find tiles with wrong annotations.
Thus, the percentage of wrong annotations corrected does not attain very high
values. When the annotations are of bad quality (e.g., 40% of simulated
additions and removals) it is easier to find the tiles with wrong annotations
and the percentage of wrong annotations corrected is very high, at the cost of
analyzing around 20% of the tiles in the Zimbabwe dataset.

In general, when the probability maps are accurate higher values of $r_{k}$
can be used and therefore less amount of tiles will be analyzed by user
annotators. On the other hand, when the probability maps are not reliable
lower values of $r_{k}$ should be used to find most of the tiles with wrong
annotations at the cost of correcting/verifying more tiles.

Tanzania dataset
---
|
(a) | (b)
Zimbabwe dataset
|
(c) | (d)
Figure 8: Stopping criteria performance using simulated addition, removals and
small shifts.

###  4.5 Experiments with real users

Table 3 presents the results showing the percentage of wrong annotations
corrected and percentages of tiles analyzed for different values of $r_{k}$
(stopping criteria parameter) in the two analyzed datasets. For the Mugumu
dataset, we can observe that almost all the wrong annotations were corrected
when the annotation process stops using the three values of $r_{k}$. The
results in the Gweru are similar, but considerably more samples are analyzed
when $r_{k}=0.02$ than when $r_{k}=0.10$. These results using real user
annotation corrections confirm the effectiveness of the proposed method in a
more realistic scenario. During the experiment, the volunteers commented that
at the beginning of the experiment most of the tiles require some annotation
correction and the need for correction decreased over time.

Table 3: Results of the experiments with real user interactions. We report the
percentage of wrong annotations corrected (% corrected) and percentages of
tiles analyzed for three different values of $r_{k}$ (stopping criteria
parameter) in the Mugumu and Gweru datasets. Experiments with real user
interactions
---
$r_{k}$ (%) | Mugumu dataset (Tanzania) | Gweru dataset (Zimbabwe)
| % corrected | % tiles analyzed | % corrected | % tiles analyzed
10 | 99.0 | 22.7 | 97.9 | 23.7
5 | 99.0 | 23.6 | 98.6 | 24.8
2 | 99.0 | 24.6 | 98.6 | 28.7

##  5 Conclusions

In this work, we proposed a methodology for the interactive
correction/verification of rural building annotations in OpenStreetMap, which
could be useful for the validation and quality improvement of OSM data. The
proposed methodology aims to quickly find a small number of regions that need
to be verified by a user, avoiding the task of exhaustively verifying the
large imagery obtained from the analyzed geographical area. In order to
analyze such large images, we proposed an efficient CNN building segmentation
method to obtain a building probability map. We evaluated several strategies
to measure how wrong the current annotations are, based on both the current
annotations and the generated building probability map. The sum of absolute
differences between the original annotations and the probability map leads to
better performances, among the evaluated tile selection strategies. We also
observed that the approach of retraining the CNN building segmentation model
with the newly annotated/verified data considerably improves the accuracy of
the method. The experiments that involve real user annotations show that the
proposed stopping criterion allows our method to analyze less than a quarter
of the total number of tiles obtained from the image and correct more than 98%
of the annotation errors. We believe that the proposed methodology could be
very helpful for improving the annotation efficiency of humanitarian mapping
projects, such as the Humanitarian OpenStreetMap Team
888https://www.hotosm.org/. The proposed methodology not only makes more
efficient the process of annotation but also produces a better experience for
the volunteers, keeping high the levels of engagement (i.e., avoiding the
analysis of uninteresting regions). As future work, we will propose a model
that can estimate the number of edits needed to correct a group of annotations
and also analyze the annotation skill of different users.

## Acknowledgment

The authors would like to thank Bing maps and OpenStreetMap for the access to
the imagery and geographical objects’ footprints respectively through their
APIs. This research was funded by Fundação de Amparo à Pesquisa do Estado de
São Paulo (FAPESP, grant 2016/14760-5 and 2014/12236-1), Conselho Nacional de
Desenvolvimento Científico e Tecnológico (CNPq, grant 303808/2018-7),
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES,
finance code 001) and by the Swiss National Science Foundation (grant
PP00P2-150593).

## Data and codes availability statement

The data and codes that support the findings of this study are available in
figshare.com with the identifier(s) at the link
https://figshare.com/s/1147e0068fd9af8b1412. Bing imagery cannot be made
publicly available because of Bing’s imagery data policies.

## References

  * Ali _et al._ (2017) Ali, A.L., _et al._ , 2017. Rule-guided human classification of volunteered geographic information. _ISPRS Journal of Photogrammetry and Remote Sensing_ , 127, 3–15.
  * Ali _et al._ (2014) Ali, A.L., _et al._ , 2014. Ambiguity and plausibility: managing classification quality in volunteered geographic information. _In_ : _Proceedings of the 22nd ACM SIGSPATIAL international conference on advances in geographic information systems_. 143–152.
  * Alsaade (2012) Alsaade, F., 2012. Fast and accurate template matching algorithm based on image pyramid and sum of absolute difference similarity measure. _Research Journal of Information Technology_ , 4 (4), 204–211.
  * Audebert _et al._ (2017) Audebert, N., Saux, B.L., and Lefèvre, S., 2017. Joint learning from Earth Observation and OpenStreetMap data to get faster better semantic maps. _In_ : _Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_.
  * Chen _et al._ (2018) Chen, J., _et al._ , 2018. Deep learning from multiple crowds: A case study of humanitarian mapping. _IEEE Transactions on Geoscience and Remote Sensing_ , 1–10.
  * Chen and Zipf (2017) Chen, J. and Zipf, A., 2017. DeepVGI: Deep Learning with Volunteered Geographic Information. _In_ : _Proceedings of the 26th International Conference on World Wide Web Companion_. 771–772.
  * Cheng _et al._ (2019) Cheng, D., _et al._ , 2019. DARNet: Deep Active Ray Network for Building Segmentation. _In_ : _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7431–7439.
  * Demir _et al._ (2018) Demir, I., _et al._ , 2018. Deepglobe 2018: A challenge to parse the earth through satellite images. _In_ : _Computer Vision and Pattern Recognition Workshops (CVPRW)_.
  * Fleischmann _et al._ (2017) Fleischmann, P., _et al._ , 2017. Using OpenStreetMap for autonomous mobile robot navigation. _In_ : _Intelligent Autonomous Systems 14_.
  * Gomes _et al._ (2011) Gomes, R., _et al._ , 2011. Crowdclustering. _In_ : _Proceedings of Advances in Neural Information Processing Systems (NIPS)_.
  * Hamaguchi and Hikosaka (2018) Hamaguchi, R. and Hikosaka, S., 2018. Building detection from satellite imagery using ensemble of size-specific detectors. _In_ : _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 223–2234.
  * He _et al._ (2017) He, K., _et al._ , 2017. Mask R-CNN. _In_ : _Proceedings of the IEEE international conference on computer vision_. 2961–2969.
  * Jilani _et al._ (2014) Jilani, M., Corcoran, P., and Bertolotto, M., 2014. Automated highway tag assessment of OpenStreetMap road networks. _In_ : _Proceedings of the 22Nd ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , SIGSPATIAL ’14. 449–452.
  * Kaiser _et al._ (2017) Kaiser, P., _et al._ , 2017. Learning aerial image segmentation from online maps. _IEEE Transactions on Geoscience and Remote Sensing_ , 55 (11), 6054–6068.
  * Liu _et al._ (2018) Liu, S., _et al._ , 2018. Path aggregation network for instance segmentation. _In_ : _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8759–8768.
  * Luxen and Vetter (2011) Luxen, D. and Vetter, C., 2011. Real-time routing with OpenStreetMap data. _In_ : _Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , GIS ’11. 513–516.
  * Maggiori _et al._ (2017) Maggiori, E., _et al._ , 2017. Convolutional neural networks for large-scale remote-sensing image classification. _IEEE Transaction on Geoscience and Remote Sensing_ , 55 (2), 645–657.
  * Marcos _et al._ (2018) Marcos, D., _et al._ , 2018. Learning deep structure active contours end-to-end. _In_ : _Computer Vision and Pattern Recognition (CVPR)_.
  * Matasci _et al._ (2012) Matasci, G., Tuia, D., and Kanevski, M., 2012. Svm-based boosting of active learning strategies for efficient domain adaptation. _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 5 (5), 1335–1343.
  * Mnih and Hinton (2012) Mnih, V. and Hinton, G.E., 2012. Learning to label aerial images from noisy data. _In_ : _International Conference on Machine Learning_. 567–574.
  * Neis and Zielstra (2014) Neis, P. and Zielstra, D., 2014. Recent developments and future trends in volunteered geographic information research: The case of OpenStreetMap. _Future Internet_ , 6 (1), 76–106.
  * Pluim _et al._ (2003) Pluim, J.P.W., Maintz, J.B.A., and Viergever, M.A., 2003. Mutual-information-based registration of medical images: a survey. _IEEE transactions on medical imaging_ , 22 (8), 986–1004.
  * Ronneberger _et al._ (2015) Ronneberger, O., Fischer, P., and Brox, T., 2015. U-net: Convolutional networks for biomedical image segmentation. _In_ : _International Conference on Medical image computing and computer-assisted intervention_. 234–241.
  * Saito _et al._ (2016) Saito, S., Yamashita, T., and Aoki, Y., 2016. Multiple object extraction from aerial imagery with convolutional neural networks. _Electronic Imaging_ , 2016 (10), 1–9.
  * Srivastava _et al._ (2018) Srivastava, S., _et al._ , 2018. Fine-grained landuse characterization using ground-based pictures: a deep learning solution based on globally available data. _International Journal of Geographical Information Science_ , 1–20.
  * Srivastava _et al._ (2019) Srivastava, S., Vargas-Muñoz, J.E., and Tuia, D., 2019. Understanding urban landuse from the above and ground perspectives: A deep learning, multimodal solution. _Remote Sensing of Environment_ , 228, 129 – 143.
  * Tasar _et al._ (2018) Tasar, O., _et al._ , 2018. Polygonization of binary classification maps using mesh approximation with right angle regularity. _In_ : _IEEE International Geoscience and Remote Sensing Symposium (IGARSS)_ , 7. 6404–6407.
  * Vargas-Muñoz _et al._ (2019) Vargas-Muñoz, J.E., _et al._ , 2019. Correcting rural building annotations in OpenStreetMap using convolutional neural networks. _ISPRS Journal of Photogrammetry and Remote Sensing_ , 147, 283 – 293.
  * Volpi and Tuia (2017) Volpi, M. and Tuia, D., 2017. Dense semantic labeling of subdecimeter resolution images with convolutional neural networks. _IEEE Transactions on Geoscience and Remote Sensing_ , 55 (2), 881–893.
  * Wang _et al._ (2017) Wang, S., _et al._ , 2017. TorontoCity: Seeing the World With a Million Eyes. _In_ : _Proceedings of the IEEE International Conference on Computer Vision_. 3009–3017.
  * Wang and Zipf (2017) Wang, Z. and Zipf, A., 2017. Using OpenStreetMap Data to Generate Building Models with Their Inner Structures for 3d Maps. _ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , 411–416.