# LiSA: A Lightweight and Secure Authentication Mechanism for Smart Metering
Infrastructure

Sahil Garg1, Member, IEEE, Kuljeet Kaur1, Member, IEEE, Georges Kaddoum1,
Member, IEEE,
François Gagnon1, Senior Member, IEEE, Syed Hassan Ahmed2, Senior Member,
IEEE,
Dushantha Nalin K. Jayakody3, Senior Member, IEEE (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS> 1Electrical Engineering
Department, École de technologie supérieure, Montréal, QC H3C 1K3, Canada.
2Department of Computer Science, Georgia Southern University, Statesboro, GA
30460, USA.
3School of Computer Science and Robotics, National Research Tomsk Polytechnic
University, Russia.

###### Abstract

Smart metering infrastructure (SMI) is the core component of the smart grid
(SG) which enables two-way communication between consumers and utility
companies to control, monitor, and manage the energy consumption data. Despite
their salient features, SMIs equipped with information and communication
technology are associated with new threats due to their dependency on public
communication networks. Therefore, the security of SMI communications raises
the need for robust authentication and key agreement primitives that can
satisfy the security requirements of the SG. Thus, in order to realize the
aforementioned issues, this paper introduces a lightweight and secure
authentication protocol, “LiSA”, primarily to secure SMIs in SG setups. The
protocol employs Elliptic Curve Cryptography at its core to provide various
security features such as mutual authentication, anonymity, replay protection,
session key security, and resistance against various attacks. Precisely, LiSA
exploits the hardness of the Elliptic Curve Qu Vanstone (EVQV) certificate
mechanism along with Elliptic Curve Diffie Hellman Problem (ECDHP) and
Elliptic Curve Discrete Logarithm Problem (ECDLP). Additionally, LiSA is
designed to provide the highest level of security relative to the existing
schemes with least computational and communicational overheads. For instance,
LiSA incurred barely 11.826 ms and 0.992 ms for executing different passes
across the smart meter and the service providers. Further, it required a total
of 544 bits for message transmission during each session.

###### Index Terms:

Authentication protocol, Elliptic Curve Cryptography, Smart Metering
Infrastructure, Smart Grid, and Security features.

##  I Introduction

Smart Grid (SG) is the next-generation power system that greatly enhances the
reliability, efficiency and sustainability of the legacy power systems with
renewable energy sources, distributed intelligence, and improved demand
response features [1, 2]. As opposed to the traditional grid, SG allows a bi-
directional flow of energy and information in order to enable new
functionalities among the consumers and utilities. The smart metering
infrastructure (SMI) is one of the crucial application domains of SGs that
helps in evaluating the status of a power grid along with the management of
the distributed resources [3, 4]. In SGs, the smart meters (SMs) connect power
consumers to the utility company in order to exchange, manage and control the
energy consumption. They rely on advanced information and communication
technologies to support the intelligent power supply and enhance the
efficiency of legacy power systems[5, 6]. However, due to the lack of inherent
and effective security mechanisms, interactions between legal entities are
susceptible to cyber-attacks [7, 8]. Nevertheless, the complex nature of the
SG and its diverse security requirements pose challenges to its widespread
adoption. Thus, it is necessary to provide secure and reliable authentication
systems for SMs that not only maintain trust between the legitimate entities
but also satisfy other security services like mutual authentication,
integrity, and anonymity [9, 10].

In order to address the privacy issues of SMs, several authentication schemes
have recently been proposed. For example, Diao et al. [11] introduced a
privacy preserving smart metering protocol that employs Camenisch–Lysyanskaya
signature scheme for providing a secure and reliable authentication system.
This scheme constructs linkable anonymous credential in order to provide
message authentication and traceability of faulty SMs. In a similar context,
Yu et al. [12] designed an information centric networking (ICN) based
approach; wherein a novel key management scheme was employed in order to
ensure the security in SMI. Similarly, Abbasinezhad-Mood and Nikooghadam [13]
devised a lightweight communication scheme for SMs which employs the hash and
exclusive-OR operations in order to provide confidentiality, real-time
authentication along with the ability to cope with one-minute or even less
time intervals of data transmission. Likewise, the authors in [14] employed an
identity-based non-interactive key distribution scheme to develop a secure and
scalable key distribution framework for neighboring SMs. In order to provide
compression and authentication of smart meter readings in SMIs, Lee et al.
[15] introduced a unified approach based on the notion of compressive sensing
in a multicarrier system. Here, the residual error of a received signal was
used to determine whether the signal is legitimate or not.

In 2018, Mustapa et al. [16] proposed an authentication scheme for SMI;
wherein ring oscillator physically unclonable functions (ROPUF) were employed
for deriving and storing the cryptographic keys. In [17], Mohammadali et al.
utilized Elliptic Curve Cryptography (ECC) to propose an identity-based key
establishment protocol for SMIs. Likewise, Kumar et al. [18] presented a
lightweight authentication and key agreement scheme in order to maintain the
privacy in SMIs. In an another work, Abbasinezhad-Mood and Nikooghadam [19]
developed an anonymous ECC-based self-certified key distribution scheme for
SGs in order to solve the issues of public key infrastructure (PKI)
maintenance and the key escrow problem. Similarly, Braeken et al. [20] also
presented a key agreement model for smart metering communications which aims
to provide session key security under the widely accepted Canetti-Krawczyk
security model. The authors asserted that their scheme provides identity-based
mutual authentication, credential privacy, and session key security, along
with the resistance against the well-known attacks such as replay, man-in-the-
middle, and impersonation. Although several authentication and key exchange
protocols exist in the literature to provide security in SGs, most of them not
only fail to provide SM anonymity but also require high communicational and
computational costs. Thus, in this paper a Lightweight and Secure
Authentication (LiSA) protocol is designed for securing SMIs.

###  I-A Contributions

The key contributions of this paper are illustrated below:

  * •

We propose a provably secure mutual authentication and key agreement protocol
named LiSA, specifically for SMIs employed in SG environments. The designed
protocol leverages the hardness of the Elliptic Curve Qu Vanstone (EVQV)
certificate mechanism, Elliptic Curve Diffie Hellman Problem (ECDHP), and
Elliptic Curve Discrete Logarithm Problem (ECDLP) in its design to prevent
various security attacks.

  * •

Additionally, we provide a detailed informal security evaluation of the
proposed protocol. The evaluation reveals that LiSA supports mutual
authentication, anonymity, replay protection, session key security, and
resists various other attacks.

  * •

The proposed protocol is a perfect mix of security features and lightweight
cryptographic functions that make it apt for deployment in SG ecosystems.
Thus, in order to validate it efficacy it has been extensively evaluated
against the current state-of-the-art in terms of security features,
computational complexity, and communicational complexity.

###  I-B Organization

The present manuscript is structured according to the following sequence. The
proposed system model is sketched in Section II and the designed protocol is
presented in Section III. Following this, detailed security analysis and
performance assessment of the proposed scheme are demonstrated in Sections IV
and V, respectively. Lastly, the conclusions are drawn in Section VI.

##  II System Model

A typical schematic diagram of the proposed LiSA protocol is depicted in Fig.
1. The setup comprises primarily of three entities, i.e., SMs, Service
Providers (SPs), and Trusted Third Party (TTP). SMs are resource constrained
electronic devices deployed across customers’ premises to record energy
consumption on a periodic basis. Additionally, SMs are also responsible for
collecting and transmitting the energy consumption data to the utility. On the
other hand, SPs are utilities that provide services to the end-users. Above
all, TTP is the trusted entity that initializes the system parameters and aids
in registration of SMs and SPs. Following this, mutual authentication and key
exchange is executed between the SMs and SPs. The execution details of LiSA
are elaborated in the upcoming segment.

Figure 1: A typical setup of LiSA.

##  III LiSA: A Mutual Authentication and Key Exchange Strategy for SMI

In the proposed scheme, the concepts of ECQV, ECDLP, and ECDHP are exploited
to establish trust and authenticity between the SMs and SPs to resist
different attack vectors. The overall execution strategy has been segregated
into the following broad phases: i) Setup Phase, ii) Registration Phase, and
iii) Authentication and Key Exchange Phase [21, 22]. In summary, the by-
product of the protocol is equivalent to “mutual authentication”, while the
end-product is the “exchange of a secure session key”. The detailed
information is given below:

###  III-A Phase I: Setup Phase

During this phase, the SMI is prepared for the subsequent phases and
incorporates the following course of actions:

Step 1: It begins with the selection of an elliptic curve $E$ by the TTP. The
$E$ is further associated with a generator $P$ and order $q$.

Step 2: Following this, the TTP computes its private ($d_{T}$) and public key
($Q_{T}$) pairs. This is done in accordance with the following equations:
$d_{T}\in Z^{*}_{q}$ and $Q_{T}=d_{T}.P$.

Step 3: Next, a one-way hash function, i.e., ($H_{0}()$) is selected.

Step 4: Finally, the above mentioned parameters including
$<E,P,q,H_{0}(),Q_{T}>$ are made public.

###  III-B Phase II: Registration Phase

During this phase, the SMs and the SPs get themselves registered with the TTP.
The process of registration for both the entities is exactly alike and is
based on the ECQV certificate scheme [20]. Thus, it is discussed in the case
of SMs only, but is extendable to the SPs as well. The execution steps of the
registration process are detailed in Fig. 2 and discussed as under:

Step 1: The $A^{th}$ SM starts its registration process by choosing an
identity ($ID_{A}$) for its itself. Next, it selects a random number ($r_{A}$)
and computes $R_{A}$. Finally, the SM transmits $<ID_{u},~{}R_{A}>$ to the TTP
using the secure channel.

Step 2: Upon receiving $<ID_{u},~{}R_{A}>$, the TTP selects a random number
$r_{T}$ and computes $R_{T}$. It then computes a certificate for the $A^{th}$
SM using the following operation: $Cert_{A}=R_{A}+R_{T}$. Following this, the
TTP calculates the value of variable $r$ using $H_{0}()$, concatenation, and
ECC point addition operations.

Step 3: Finally, the computed value of $Cert_{A}$ and $r$ are sent to the
$A^{th}$ SM over the secure channel.

Step 4: Using the received values, the SM computes its private key ($d_{A}$)
followed by its public key $Q_{A}$.

Smart Meter ($SM_{A}$) |  | Trusted Third Party
---|---|---
$\bullet$Select $ID_{A}$
$\bullet$Select $r_{A}\in Z^{*}_{q}$
$\bullet$Compute $R_{A}=r_{A}.P$
$\xrightarrow{<ID_{A},~{}R_{A}>~{}\text{over secure channel}}$
|  $\bullet$Generate $r_{T}\in Z^{*}_{q}$
|  $\bullet$Compute $R_{T}=r_{T}.P$
|  $\bullet$Compute $Cert_{A}=R_{A}+R_{T}$
|  $\bullet$Compute $r=H_{0}(Cert_{A}||ID_{A})r_{T}+d_{T}$
$\xleftarrow{<Cert_{A},~{}r>~{}\text{over secure channel}}$
$\bullet$Compute $d_{A}=H_{0}(Cert_{A}||ID_{A})r_{A}+r$
$\bullet$Compute $Q_{A}=d_{A}P$
$=H_{0}(Cert_{A}||ID_{A})Cert_{A}+Q_{T}$
Figure 2:  Phase II: Registration Phase.

###  III-C Phase III: Authentication & Key Exchange Phase

This is the most crucial phase of LiSA and incorporates mutual authentication
and session key exchange between the SMs and SPs. The series of execution
steps are reflected in Fig. 3 and explained as follows.

Step 1: The process is initiated by the $A^{th}$ SM by generating its current
time stamp ($T_{SM}$) and a random number ($r_{SM}$). Next, it computes
$R_{SM}$ and $R_{SM}^{{}^{\prime}}$ using ECC point multiplication.

Step 2: Following this, the $A^{th}$ SM computes an intermediate token
($\mathbb{T}o_{SM}$) using its $Cert_{A}$ and $ID_{A}$. It also calculates an
authorization token ($\mathbb{A}uth_{SM}$) for the SP to verify using the
following operation:
$\mathbb{A}uth_{SM}=R_{SM}^{{}^{\prime}}\oplus\mathbb{T}o_{SM}\oplus T_{SM}$.

Step 3: Next, $<T_{SM},~{}R_{SM},~{}\mathbb{A}uth_{SM}>$ are transmitted to
$SP_{B}$ over the open channel. It is worth noting here that the SM’s ID is
never transmitted over the channel in clear text format which prevents
tracking of SM by the adversary $\mathcal{A}$.

Step 4: Once $<T_{SM},~{}R_{SM},~{}\mathbb{A}uth_{SM}>$ are received by the
SP, it validates the time stamp $T_{SM}$. If found valid, the SP proceeds;
else the connection is terminated immediately. Next, the SP computes
$R_{SM}^{{}^{\prime\prime}}=d_{B}.R_{SM}$; followed by the extraction of
$(Cert_{A}||ID_{A})$ from the received $\mathbb{A}uth_{SM}$. Using the
extracted values, the SP computes $SM_{A}$’s public key:
$Q_{A}=H_{0}(Cert_{A}||ID_{A})Cert_{A}+Q_{T}$. If the computed key matches the
publicly available $Q_{A}$, then the SP is sure about the authenticity of the
SM. Otherwise, the connection is dropped by the SP.

Step 5: Next, the SP records its time stamp $T_{SP}$ and generates another
authentication token ($\mathbb{A}uth_{SP}$) for the SM to verify.
Simultaneously, the SP also computes a session key ($SK_{B-A}$) that is used
only if the mutual authentication is established. $SK_{B-A}$ is computed using
the following operation:
$kdf(ID_{A}||Cert_{A}||R_{SM}^{{}^{\prime\prime}}||T_{SP}||T_{SM})$ where
$kdf$ is a key derivative function.

Step 6: Finally, the parameters $<T_{SP},~{}\mathbb{A}uth_{SP}>$ are
communicated to the SM for the next course of action.

Step 7: On receiving the above mentioned parameters, the validation of time
stamp $T_{SP}$ is done. Next, the authentication token
$\mathbb{A}uth_{SP}^{*}$ is computed for verifying it against the received
token. A match indicates that the SP is a valid/authentic entity.

Step 8: Finally, the SM computes the shared session key as follows:
$SK_{A-B}=kdf(ID_{A}||Cert_{A}||R_{SM}^{{}^{\prime}}||T_{SP}||T_{SM}$.

Smart Meter $SM_{A}$ |  Service Provider $SP_{B}$
---|---
$\bullet$Generate time stamp $T_{SM}$
$\bullet$Select a random number $r_{SM}\in Z^{*}_{q}$
$\bullet$Compute $R_{SM}=r_{SM}.P$
$\bullet$Compute $R_{SM}^{{}^{\prime}}=r_{SM}.Q_{B}$
$\bullet$Compute $\mathbb{T}o_{SM}=(Cert_{A}||ID_{A})$
$\bullet$Compute
$\mathbb{A}uth_{SM}=R_{SM}^{{}^{\prime}}\oplus\mathbb{T}o_{SM}\oplus T_{SM}$
$\xrightarrow{<T_{SM},~{}R_{SM},~{}\mathbb{A}uth_{SM}>}$
|  |  $\bullet$Validate time stamp $T_{SM}$
|  |  $\bullet$Compute $R_{SM}^{{}^{\prime\prime}}=d_{B}.R_{SM}$
|  |  $\bullet$Extract $(Cert_{A}||ID_{A})=\mathbb{A}uth_{SM}\oplus R_{SM}^{{}^{\prime\prime}}\oplus T_{SM}$
|  |  $\bullet$Compute $Q_{A}=H_{0}(Cert_{A}||ID_{A})Cert_{A}+Q_{T}$
|  |  $\bullet$Check $Q_{A}\stackrel{{\scriptstyle?}}{{=}}\text{Public Key of SM with}~{}ID_{A}$
|  |  If same, $SM_{A}$ is authentic; else disconnect
|  |  $\bullet$Generate time stamp $T_{SP}$
|  |  $\bullet$Compute $\mathbb{A}uth_{SP}=H_{0}((R_{SM}^{{}^{\prime\prime}}||Cert_{A}||ID_{A})$
|  |  $\oplus~{}T_{SP})$
|  |  $\bullet$Compute $SK_{B-A}=$
|  | $kdf(ID_{A}||Cert_{A}||R_{SM}^{{}^{\prime\prime}}||T_{SP}||T_{SM})$
$\xleftarrow{<T_{SP},~{}\mathbb{A}uth_{SP}>}$
$\bullet$Validate time stamp $T_{SP}$
$\bullet$Compute
$\mathbb{A}uth_{SP}^{*}=H_{0}((R_{SM}^{{}^{\prime}}||Cert_{A}||ID_{A})\oplus
T_{SP})$
$\bullet$Check
$\mathbb{A}uth_{SP}^{*}\stackrel{{\scriptstyle?}}{{=}}\mathbb{A}uth_{SP}$
If same, SP is authentic; else disconnect
$\bullet$Compute
$SK_{A-B}=kdf(ID_{A}||Cert_{A}||R_{SM}^{{}^{\prime}}||T_{SP}||T_{SM})$
Figure 3: Phase III: Authentication & Key Exchange Phase.

##  IV Security Analysis and Comparisons

In this section, detailed security analysis of the proposed LiSA protocol is
illustrated. Along with this, the comparative security assessment against the
existing schemes is also presented herewith.

###  IV-A Security Analysis

LiSA provides a number of security features and resists various attack
vectors. The detailed information is illustrated below.

####  IV-A1 Mutual authentication

LiSA supports the concept of mutual authentication between the communication
parties, i.e., SMs and SPs. The process of mutual authentication helps to
establish the authenticity of one party to the other. In the designed
protocol, it is realised using two different intermediate authentication
tokens namely $\mathbb{A}uth_{SM}$ and $\mathbb{A}uth_{SP}$. The hardness of
these tokens can be attributed to ECQV and ECDHP [20]. In detail, these tokens
cannot be computed without the knowledge of $Cert_{A},ID_{A},$ and $d_{B}$;
which are specifically private to the legitimate entities and are not
shareable.

####  IV-A2 Replay protection

Resistance against replay attacks enable the protocol to deny any previous
message to retransmitted between the legitimate parties. In the proposed LiSA
protocol, replay protection has been provided using the concept of time stamps
and pseudo random numbers. These time stamps and numbers enable new values to
the intermediate tokens in every session. Further, during every session, the
time stamps are verified for their validity. In case of violation, the ongoing
communications are terminated to avoid replay attacks.

####  IV-A3 Resistance against impersonation attacks

The proposed protocol also resists impersonation attacks. This implies that an
adversary $\mathbb{A}$ cannot impersonate a valid SM to form a legitimate
session with the SP. This can be credited to the strength of the ECQV
certificate mechanism and the $ID$ masking mechanism that prevents an
$\mathbb{A}$ to track SMs and spoof their identities. Further, extraction of a
SM’s certificate and identity from $\mathbb{A}uth_{SM}$ is an intractable
process for an $\mathbb{A}$. This is because, these values can only be
extracted by a valid SP (say $SP_{B}$ with which the SM is trying to connect)
with its respective private key $d_{B}$.

####  IV-A4 Resistance against man-in-the-middle (MITM) attacks

The MITM attack allows an $\mathbb{A}$ to intercept a legitimate session and
spoof the identity of an entity. Following this, $\mathbb{A}$ alters the
relayed messages to illegitimately develop a session with the SM/SP. However,
as explained earlier, LiSA resists impersonation and replay attacks. Further,
the alteration of $\mathbb{A}uth_{SM}$ and $\mathbb{A}uth_{SP}$ tokens is also
not possible in the proposed scheme. Thus, LiSA prevents MITM attacks.

####  IV-A5 Supports anonymity

LiSA also support perfect anonymity for the participating SMs. This is
achieved by masking the SM’s identity and certificate information while trying
to develop a session with the SPs. Precisely, the SM’s confidential
information such as $Cert_{A}$ and $ID_{A}$ are never relayed in clear text
format. Further, their extraction from the intermediate tokens by an
illegitimate party is an intractable process. This can be attributed to the
hardness of ECDLP [20].

####  IV-A6 Session key security

The session keys ($SK_{A-B}$ and $SK_{B-A}$) derived by the SMs and SPs in
LiSA are completely secure. The session keys can only be computed by the valid
and authentic parties post successful mutual authentication as discussed in
Section III-C.

####  IV-A7 Resistance against Denial of Service (DoS) attacks

In LiSA, the process of mutual authentication is essentially initiated by the
SM by relaying the following message: $\textless
T_{SM},~{}R_{SM},~{}\mathbb{A}uth_{SM}\textgreater$. Post receiving this
message, the SP makes checks i) the validity of the received time stamp
$T_{SM}$ followed by ii) the authenticity of the SM using
$\mathbb{A}uth_{SM}$. In the course of this process, if any validation check
fails, then the SP knows that the other party is illegitimate. Thus, the
connection is immediately dropped to prevent DoS attacks.

###  IV-B Security Comparison

This sections presents the detailed comparison between the proposed LiSA
protocol and the existing schemes [23, 24, 19, 20]. The comparison is carried
out on the basis of following features: mutual authentication
($\mathfrak{SF}_{1}$), replay protection ($\mathfrak{SF}_{2}$), resistance
against impersonation attacks ($\mathfrak{SF}_{3}$), resistance against MITM
attacks ($\mathfrak{SF}_{4}$), anonymity ($\mathfrak{SF}_{5}$), session key
security ($\mathfrak{SF}_{6}$), resistance against DoS attacks
($\mathfrak{SF}_{7}$), and lightweight ($\mathfrak{SF}_{8}$).

TABLE I: Analysis of security features. Protocol |  [23] |  [24] |  [19] |  [20] |  Proposed
---|---|---|---|---|---
$\mathfrak{SF}_{1}$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\times$ |  $\checkmark$
$\mathfrak{SF}_{2}$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{3}$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{4}$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{5}$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{6}$ |  $\times$ |  $\times$ |  $\times$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{7}$ |  $\times$ |  $\times$ |  $\times$ |  $\checkmark$ |  $\checkmark$
$\mathfrak{SF}_{8}$ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  $\checkmark$

It is evident from the comparison shown in Table I that the proposed LiSA
protocol is the most secure out of the existing schemes. For instance, Odelu
et al.’s [23], Chen et al.’s [24], and Abbasinezhad-Mood & Nikooghadam’s
schemes [19] fail to provide session key security and resistance against DoS
attacks. Additionally, these protocols are not lightweight in comparison to
the other protocols. Finally, Braeken et al.’s scheme [20] neither provides
mutual authentication nor is lightweight.

##  V Performance Evaluation

In this section, the comparative analysis of the proposed LiSA protocol is
carried out against the current state-of-the-art [23, 24, 19, 20]. The related
comparisons have been performed on the basis of total computational and
communicational complexities associated with these protocols. The detailed
information about these assessments and the related simulation parameters are
discussed herewith.

###  V-A Simulation Details

Braeken et al.in [20] adopted an efficient means to simulate the SG setup and
evaluated different protocols on it. In the said setup, the authors employed a
personal computer to mimic the computational capability of the SPs; while a
constrained device was used to represent a SM. The configurations of SPs
included Windows 7 running on 2.5 GHz CPU and 8 GB RAM. On the other hand, SMs
had limited capabilities with a single core 798 MHz CPU and 256 MB RAM.

###  V-B Evaluation Parameters

For the evaluation of the proposed LiSA protocol in comparison to the existing
protocols, the following metrics have been chosen:

####  V-B1 Computational Overhead Analysis

The computational time of the proposed and existing protocols have been
computed by taking into consideration the number of computationally expensive
cryptographic operations executed during the authentication and key exchange
phases. Table II illustrates the computational time (in ms) required for
executing different functions on the SMs and SPs, respectively. Here,
$\mathbb{T}_{b},\mathbb{T}_{m},\mathbb{T}_{a},\mathbb{T}_{h}$,
$\mathbb{T}_{e}$, and $\mathbb{T}_{s}$ refer to the execution time for
performing bilinear pairing, point multiplication, point addition, hashing,
modular exponentiation operation, and symmetric encryption/decryption
operation, respectively.

TABLE II: Execution time of different cryptographic functions. Entity |  $\mathbb{T}_{b}$ |  $\mathbb{T}_{m}$ |  $\mathbb{T}_{a}$ |  $\mathbb{T}_{h}$ |  $\mathbb{T}_{e}$ |  $\mathbb{T}_{s}$
---|---|---|---|---|---|---
SP |  17.01 |  0.986 |  0.004 |  0.001 |  0.874 |  0.033
SM |  9.23 |  5.9 |  0.004 |  0.026 |  7.86 |  0.079

In accordance with the values depicted in the above table, the computational
complexities of different protocols have been computed. The related outcomes
are shown Table III; wherein the overheads on the SMs and SPs are depicted
separately. The proposed LiSA protocol requires 11.826 msec on the SM level
and almost 0.992 msec on the SP level, for executing the authentication
mechanism along with the key exchange. Additionally, the said execution time
is the least in comparison with the other protocols as evidenced from the
results shown in Table III.

TABLE III: An illustration of computational complexity analysis. Protocol |  At SM |  At SP
---|---|---
Odelu et al.[23] |  $3\mathbb{T}_{m}+\mathbb{T}_{a}+\mathbb{T}_{e}+6\mathbb{T}_{h}\approx 25.72$ |  $2\mathbb{T}_{m}+\mathbb{T}_{a}+2\mathbb{T}_{b}+\mathbb{T}_{e}+6\mathbb{T}_{h}\approx 37.28$
Chen et al.[24] |  $2\mathbb{T}_{m}+\mathbb{T}_{e}+5\mathbb{T}_{h}\approx 19.79$ |  $3\mathbb{T}_{m}+\mathbb{T}_{b}+\mathbb{T}_{e}+5\mathbb{T}_{h}\approx 21.26$
Abbasinezhad-Mood [19] |  $4\mathbb{T}_{m}+\mathbb{T}_{a}+5\mathbb{T}_{h}\approx 23.80$ |  $4\mathbb{T}_{m}+\mathbb{T}_{a}+5\mathbb{T}_{h}\approx 3.98$
Braeken et al.[20] |  $4\mathbb{T}_{m}+\mathbb{T}_{a}+\mathbb{T}_{s}+5\mathbb{T}_{h}\approx 23.81$ |  $4\mathbb{T}_{m}+2\mathbb{T}_{a}+\mathbb{T}_{s}+5\mathbb{T}_{h}\approx 3.99$
Proposed |  $2\mathbb{T}_{m}+\mathbb{T}_{h}\approx 11.826$ |  $\mathbb{T}_{m}+\mathbb{T}_{a}+2\mathbb{T}_{h}\approx 0.992$

####  V-B2 Communicational Overhead Analysis

In order to analyse the communicational complexity of the proposed and the
existing protocols, the number of messages and bits transmitted between the
SMs and SPs have been considered. For this analysis, the transmitted messages
have been segregated into different tokens with variable lengths. For
instance, the length of the hash function and random numbers is equivalent to
160 bits, while the length of the identity field and time stamps are set to 32
bits each. Further, the x-coordinate of an ECC point requires 60 bits.
Likewise, the message length of other tokens can be found in the literature
[20]. Consequently, the communicational overheads associated with the
transmission of different tokens across the authentication and key exchange
mechanism are summarized in Table IV. It is quite clear from the obtained
results that LiSA requires the least number of rounds and number of bits to
establish trust, authenticity and secure session key between the communicating
parties. In detail, LiSA requires 2 rounds and 544 bits for transmission of
messages between the SMs and SPs.

Thus, it can be summarized from the above discussion that LiSA is the most
secure and lightweight authentication scheme for SMIs in SG setups.

TABLE IV: An illustration of communicational complexity analysis. Protocol |  Number of Rounds |  No. of bits
---|---|---
Odelu et al.[23] |  3 |  1920
Chen et al.[24] |  3 |  1632
Abbasinezhad-Mood [19] |  3 |  832
Braeken et al.[20] |  3 |  832
Proposed |  2 |  544

##  VI Conclusion

In this paper, a provably secure mutual authentication and key agreement
protocol named LiSA was devised, specifically for SMIs employed in SG
environments. The proposed protocol is a perfect mix of security features and
lightweight cryptographic functions that make it apt for deployment in SG
ecosystems. In order to attain the above mentioned features, the designed
protocol utilizes ECC, one-way hash functions, concatenation, and logical XOR
operations. In terms of ECC, the protocol leverages the concept of ECQV,
ECDLP, and ECDHP to generate the intermediate tokens including authentication
tokens and session keys. Further, the detailed security assessment of the
protocol reveals that it supports various security features namely mutual
authentication, replay protection, anonymity, resistance against MITM, DoS,
impersonation attacks, etc. Additionally, the computational and
communicational overheads associated with the execution of the designed
protocol in SG setups indicate that it is relatively lightweight in comparison
with the current state-of-the-art.

## Acknowledgment

This work was supported by the Tier 2 Canada Research Chair.

## References

  * [1] K. Kaur, S. Garg, N. Kumar, and A. Y. Zomaya, “A Game of Incentives: An Efficient Demand Response Mechanism using Fleet of Electric Vehicles,” in _Proceedings of the 1st International Workshop on Future Industrial Communication Networks_. ACM, 2018, pp. 27–32.
  * [2] G. S. Aujla, S. Garg, S. Batra, N. Kumar, I. You, and V. Sharma, “DROpS: A Demand Response Optimization Scheme in SDN-enabled Smart Energy Ecosystem,” _Information Sciences_ , vol. 476, pp. 453–473, 2019.
  * [3] P. Kumar, Y. Lin, G. Bai, A. Paverd, J. S. Dong, and A. Martin, “Smart Grid Metering Networks: A Survey on Security, Privacy and Open Research Issues,” _IEEE Communications Surveys & Tutorials_, 2019, DOI: 10.1109/COMST.2019.2899354.
  * [4] M. R. Asghar, G. Dán, D. Miorandi, and I. Chlamtac, “Smart Meter Data Privacy: A Survey,” _IEEE Communications Surveys & Tutorials_, vol. 19, no. 4, pp. 2820–2835, 2017.
  * [5] K. Kaur, S. Garg, N. Kumar, G. S. Aujla, K. K. R. Choo, and M. S. Obaidat, “An Adaptive Grid Frequency Support Mechanism for Energy Management in Cloud Data Centers,” _IEEE Systems Journal_ , 2019, DOI: 10.1109/JSYST.2019.2921592.
  * [6] K. Kaur, S. Garg, G. Kaddoum, S. H. Ahmed, and D. N. K. Jayakody, “En-OsCo: Energy-aware Osmotic Computing Framework using Hyper-heuristics,” in _Proceedings of the ACM MobiHoc Workshop on Pervasive Systems in the IoT Era_. ACM, 2019, pp. 19–24.
  * [7] R. Chaudhary, G. S. Aujla, S. Garg, N. Kumar, and J. J. Rodrigues, “SDN-enabled Multi-Attribute-based Secure Communication for Smart Grid in IIoT Environment,” _IEEE Transactions on Industrial Informatics_ , vol. 14, no. 6, pp. 2629–2640, 2018.
  * [8] S. Finster and I. Baumgart, “Privacy-Aware Smart Metering: A Survey,” _IEEE Communications Surveys & Tutorials_, vol. 17, no. 2, pp. 1088–1101, 2015.
  * [9] S. Garg, K. Kaur, G. Kaddoum, F. Gagnon, and J. J. P. C. Rodrigues, “An Efficient Blockchain-Based Hierarchical Authentication Mechanism for Energy Trading in V2G Environment,” in _IEEE International Conference on Communications Workshops (ICC Workshops), Shanghai, China_ , May 2019.
  * [10] A. Ghosal and M. Conti, “Key Management Systems for Smart Grid Advanced Metering Infrastructure: A Survey,” _IEEE Communications Surveys & Tutorials_, pp. 1–1, 2019, DOI: 10.1109/COMST.2019.2907650.
  * [11] F. Diao, F. Zhang, and X. Cheng, “A Privacy-Preserving Smart Metering Scheme using Linkable Anonymous Credential,” _IEEE Transactions on Smart Grid_ , vol. 6, no. 1, pp. 461–467, 2015.
  * [12] K. Yu, M. Arifuzzaman, Z. Wen, D. Zhang, and T. Sato, “A Key Management Scheme for Secure Communications of Information Centric Advanced Metering Infrastructure in Smart Grid,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 64, no. 8, pp. 2072–2085, 2015.
  * [13] D. Abbasinezhad-Mood and M. Nikooghadam, “An Ultra-Lightweight and Secure Scheme for Communications of Smart Meters and Neighborhood Gateways by Utilization of an ARM Cortex-M Microcontroller,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 6, pp. 6194–6205, 2018.
  * [14] V. Seferian, R. Kanj, A. Chehab, and A. Kayssi, “Identity based Key Distribution Framework for Link Layer Security of AMI Networks,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 4, pp. 3166–3179, 2018.
  * [15] Y. Lee, E. Hwang, and J. Choi, “A Unified Approach for Compression and Authentication of Smart Meter Reading in AMI,” _IEEE Access_ , vol. 7, pp. 34 383–34 394, 2019.
  * [16] M. Mustapa, M. Y. Niamat, A. P. D. Nath, and M. Alam, “Hardware-Oriented Authentication for Advanced Metering Infrastructure,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 2, pp. 1261–1270, 2018.
  * [17] A. Mohammadali, M. Sayad Haghighi, M. H. Tadayon, and A. Mohammadi-Nodooshan, “A Novel Identity-Based Key Establishment Method for Advanced Metering Infrastructure in Smart Grid,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 4, pp. 2834–2842, 2018.
  * [18] P. Kumar, A. Gurtov, M. Sain, A. Martin, and P. H. Ha, “Lightweight Authentication and Key Agreement for Smart Metering in Smart Energy Networks,” _IEEE Transactions on Smart Grid_ , vol. 10, no. 4, pp. 4349–4359, 2019.
  * [19] D. Abbasinezhad-Mood and M. Nikooghadam, “An Anonymous ECC-Based Self-Certified Key Distribution Scheme for the Smart Grid,” _IEEE Transactions on Industrial Electronics_ , vol. 65, no. 10, pp. 7996–8004, 2018\.
  * [20] A. Braeken, P. Kumar, and A. Martin, “Efficient and Provably Secure Key Agreement for Modern Smart Metering Communications,” _Energies_ , vol. 11, no. 10, p. 2662, 2018.
  * [21] K. Kaur, S. Garg, G. Kaddoum, F. Gagnon, and S. H. Ahmed, “Blockchain-Based Lightweight Authentication Mechanism for Vehicular Fog Infrastructure,” in _IEEE International Conference on Communications Workshops (ICC Workshops), Shanghai, China_ , May 2019.
  * [22] S. Garg, K. Kaur, G. Kaddoum, S. H. Ahmed, and D. N. K. Jayakody, “SDN based Secure and Privacy-preserving Scheme for Vehicular Networks: A 5G Perspective,” _IEEE Transactions on Vehicular Technology_ , 2019, DOI: 10.1109/TVT.2019.2917776.
  * [23] V. Odelu, A. K. Das, M. Wazid, and M. Conti, “Provably Secure Authenticated Key Agreement Scheme for Smart Grid,” _IEEE Transactions on Smart Grid_ , vol. 9, no. 3, pp. 1900–1910, 2018.
  * [24] Y. Chen, J.-F. Martínez, P. Castillejo, and L. López, “An Anonymous Authentication and Key Establish Scheme for Smart Grid: FAuth,” _Energies_ , vol. 10, no. 9, p. 1354, 2017.


# Leave no Place Behind: Improved Geolocation in Humanitarian Documents

Enrico M. Belliardo<EMAIL_ADDRESS>0009-0006-4456-3384 ISI
FoundationTurinItaly ,  Kyriaki Kalimeri<EMAIL_ADDRESS>ISI
FoundationTurinItaly and  Yelena Mejova<EMAIL_ADDRESS>ISI
FoundationTurinItaly

(2023)

###### Abstract.

Geographical location is a crucial element of humanitarian response, outlining
vulnerable populations, ongoing events, and available resources. Latest
developments in Natural Language Processing may help in extracting vital
information from the deluge of reports and documents produced by the
humanitarian sector. However, the performance and biases of existing state-of-
the-art information extraction tools are unknown. In this work, we develop
annotated resources to fine-tune the popular Named Entity Recognition (NER)
tools Spacy and roBERTa to perform geotagging of humanitarian texts. We then
propose a geocoding method FeatureRank which links the candidate locations to
the GeoNames database. We find that not only does the humanitarian-domain data
improves the performance of the classifiers (up to F1 = 0.92), but it also
alleviates some of the bias of the existing tools, which erroneously favor
locations in the Western countries. Thus, we conclude that more resources from
non-Western documents are necessary to ensure that off-the-shelf NER systems
are suitable for the deployment in the humanitarian sector.

Geolocation, humanitarian, location bias, geotagging, geographic bias, named
entity recognition, geocoding

††journalyear: 2023††copyright: acmlicensed††conference: ACM International
Conference on Information Technology for Social Good; September 6–8, 2023;
Lisbon, Portugal††booktitle: ACM International Conference on Information
Technology for Social Good (GoodIT ’23), September 6–8, 2023, Lisbon,
Portugal††price: 15.00††doi: 10.1145/3582515.3609515††isbn:
979-8-4007-0116-0/23/09††ccs: Social and professional topics Geographic
characteristics††ccs: Information systems Data mining††ccs: Information
systems Information systems applications

##  1\. Introduction

The vast network of humanitarian organizations performing a myriad of tasks
around the world produces a significant amount of data and reports. For
instance, the operations of the International Federation of Red Cross and Red
Crescent Societies (IFRC) spans 192 countries and includes almost 14 million
volunteers (American Red Cross, 2022). To manage the resulting reports,
projects such as the Data Entry and Exploration Platform
(DEEP)111https://thedeep.io/ have been developed to help humanitarian
organizations to compile and store documentation, and to help structure the
qualitative information inside. It is maintained by Data Friendly Space, a
U.S.-based international non-governmental organization
(INGO)222https://datafriendlyspace.org/, in collaboration with numerous
international organizations.

In the light of the information overload generated by the data-rich operation
of humanitarian efforts, computer-aided information extraction promises to
increase the usefulness of the produced documentation. Latest developments in
Deep Learning (DL) and Natural Language Processing (NLP) allow for the
identification of potentially important information in text, and its
classification in standard hierarchies, which can then be used to aggregate
knowledge and share experience across time and projects. In particular,
geolocation is an important aspect of humanitarian activity, spanning both
operations in entire countries and regions, to specific villages, refugee
camps, and civil infrastructure. The accuracy of location information is
especially important in the light of the Sustainable Development Goals (SDGs)
principle of “Leave No One Behind” that strives to reduce inequalities and
vulnerabilities amongst all countries. Unfortunately, the data sources
available for training competent models are often biased towards the West,
even when secondary data is used to fill the “datasphere” gaps (Dalton et al.,
2016). Location databases (or “gazetteers”) have a well-known bias towards the
US and other Western countries (Graham, 2015; Graham and De Sabbata, 2015),
whereas popular alternative data sources such as Twitter and Wikipedia
struggle with penetration in global South (Statista, 2022; Wikipedia, 2022).
Thus, it is imperative to create resources and tools for the accurate
processing of diverse humanitarian data in order to ensure no country is
disadvantaged when the information is collated.

In this study, we work closely with humanitarian partners to create a
Geolocation Extraction tool tuned specifically to process documents generated
by humanitarian projects. The tool performs two sub-tasks: geotagging – the
extraction of text fragments that may be a location (or “toponyms”), and
geocoding – the disambiguation of the toponym to a specific geographic
location (with accompanying information on GPS coordinates, type of location,
etc.). We contribute two datasets – one for each of these steps – wherein
humanitarian reports are annotated by specialists in the field for candidate
toponyms, and these toponyms are mapped to the appropriate entry in GeoNames,
an extensive geographical database333https://www.geonames.org/. Using these
datasets, we tune popular existing Named Entity Extraction (NER) models
Spacy444https://spacy.io/models/en#en_core_web_md and
roBERTa555https://huggingface.co/Davlan/xlm-roberta-base-wikiann-ner,
improving their performance up to F1 $=0.92$ on the target dataset. We then
propose a geocoding algorithm FeatureRank tailored to the humanitarian domain,
and compare it to two baselines from the literature. Finally, we show that the
tuned model not only improves in accuracy, but that it may alleviate the
geographic bias that favors locations in the Western countries.

##  2\. Related Literature

Named Entity Recognition (NER) is one of the basic Natural Language Processing
(NLP) tasks involving the identification of entities of interest in texts,
commonly constrained to Person, Organisation or Location. Whereas the early
models used classic machine learning algorithms including support vector
machines, random forests, and decision trees using as features capitalization,
word endings, and regular expressions (Nadeau and Sekine, 2007), from about
2011 neural networks have been used to create more generalizable models
(Collobert et al., 2011; Yadav and Bethard, 2018). Combining word context and
its individual characters has allowed the training of systems that required
little domain-specific training data. Word and sentence level embeddings have
been used to build models using several long short-term memory (LSTM) layers
to achieve performance of up to 90% F score in English (Dernoncourt et al.,
2017) (this model was also published as a NeuroNER
package666http://neuroner.com/). Graph embedding algorithms that produce
embeddings of vertices that preserve their proximity in a graph, including
DeepWalk (Perozzi et al., 2014) and TransE (Bordes et al., 2013), have been
widely used in entity linking (EL) task, the task of matching a piece of text
(a potential entity) to an entry in a knowledge base such as Freebase,
DBpedia, and Wikidata. Since 2015, large pre-trained models like Bidirectional
Encoder Representations from Transformers (BERT) (Kenton and Toutanova, 2019)
have given researchers the access to representations trained on huge amounts
of text data without the need to access this data directly, while allowing
strong expressive power suitable to many NLP tasks. These models can then be
trained using the distant labels, improving task-specific recall and
precision, and further enhanced using self-training (Liang et al., 2020).
Neural networks and pre-trained large models have also been used for entity
linking (Sevgili et al., 2022). For instance, BERT-based cross-encoders have
been used on a concatenation of text segment context with a candidate entity
description to produce a score for each entity candidate (Logeswaran et al.,
2019; Wu et al., 2020). However, few of these efforts have been directed
specifically to geographical NER.

Natural language processing approaches have been applied to understand the
context in the extraction and geocoding of historical floods, storms, and
adaptation measures (Lai et al., 2022). In the context of mass emergencies,
Imran et al. (Imran et al., 2015) highlighted the existing approaches and
challenges in processing social media messages during mass emergencies. This
survey sheds light on the importance of real-time analysis of social media
data for effective emergency response. Several techniques were explored to
extract and refine location mentions in text such as geoparsing, location
disambiguation, and geotagging (Middleton et al., 2018). For instance,
geotagging text data on the web has been approached through geometrical
methods (Radke et al., 2018), offering an alternative to relying solely on
explicit geotags. The potential of Twitter data were highlighted both in
determining the geographic origin of user-generated content (Graham et al.,
2014), and developing predictive models that can estimate the location of
Twitter users based on their posted content (Zheng et al., 2018). Moreover,
mining Twitter data offered a better understanding of disaster resilience (Zou
et al., 2018), while was also proven effective in event classification and
location prediction during disasters (Singh et al., 2019). Importantly,
efforts have been made to develop annotated datasets and resources (Poletto et
al., 2021; Imran et al., 2022) contributing to the development and evaluation
of algorithms for the identification and monitoring of internal displacement,
but also sentiment analysis, named entity recognition, and geolocation from
textual data. In this work, we contribute annotated resources concerning the
humanitarian reports, instead of general web data.

Despite these advancements there are still many open issues to accurate named
entity recognition for the humanitarian sector. For instance, known biases of
the geographic information and modes of communication are directly impacting
the representation of populations unevenly (Graham and De Sabbata, 2015).
Population bias in geotagged on social media has been a topic of concern in
geolocation research (Malik et al., 2015; Rama et al., 2020), highlighting the
potential biases that can arise from relying solely on geotagged tweets for
location inference. Understanding and addressing these biases is crucial to
ensure accurate geolocation results.

##  3\. Data collection

We employ data from the HumSet database (Fekih et al., 2022), originated from
the multi-organizational platform (DEEP) introduced earlier. Each entry
consists of a relevant excerpt from a document, annotated with the
humanitarian analysis framework categories. It is associated to a _lead_id_
that identifies the original document (_lead_) from which the excerpt is
extracted. Original documents come from different sources, including reports
by humanitarian organizations and media articles. We downloaded them using
_deepex_ package777available at https://github.com/the-deep/deepex from the
links available in HumSet. The documents come either in PDF format or HTML
pages, and text extraction from both sources is performed with the same
package. This parser also splits text in pages and paragraphs, according to
the basic structure of the document. In this project, we use the entire text
parsed from HumSet _leads_ , and not only the selection of excerpts of the
database.

###  3.1. Data description

The dataset contains $15\,661$ documents from 45 different projects. It is
multilingual, with English (67.1%), Spanish (18.6%) and French (14.1%)
representing almost the totality of the documents’ languages (language
information is available in HumSet). Each project typically focuses on one or
more countries, but some focus on none. There are 33 countries represented by
country-specific projects. The content of the documents may vary from text to
images and tables. The parser is able to fetch plain text and discard figures
and non textual elements. Some documents contain mainly non textual elements,
therefore the resulting extracted text is very short. Table 1 shows a
distribution of metrics of the text content extracted with the parser.

Table 1. Summary statistics on text files parsed from documents. | size(KB) | pages | paragraphs | words
---|---|---|---|---
mean | 11.6 | 5.0 | 38.0 | 1621
std | 40.5 | 14.5 | 130.9 | 5487
min | 0.0 | 1 | 0 | 0
25% | 1.2 | 1 | 2 | 170
50% | 3.2 | 1 | 8 | 459
75% | 7.6 | 3 | 26 | 1085
max | 1354.2 | 358 | 4254 | $207\,030$

##  4\. Data Annotation

This work contributes two annotated datasets, one for each step of the
geolocation extraction process: (1) geotagging, which is a special case of
NER, and (2) geocoding, where the found toponyms are disambiguated and linked
to specific geographic coordinates (Gritta et al., 2019). For both tasks, we
describe how the data was sampled, baseline models were applied to extract
candidates, how these candidates were annotated, guided by coding schemas.
Finally, we present the statistics about the resulting annotated datasets.

###  4.1. Annotation: geotagging

####  4.1.1. Sampling

We begin by selecting 500 English-language documents from HumSet, satisfying
several metrics, as described below. As we want our sample to include as many
different locations as possible, while respecting the country distribution
ranking of the dataset, we treat the documents belonging to the 6 most popular
project countries differently from the rest (the “tail” of the distribution).
We impose different restrictions on the popular and tail countries in terms of
on the number of pages (minimum of 3 or 2, respectively), average number of
paragraphs per page (minimum 5 or 3) and average number of words per paragraph
(minimum 500 or 300). This filter results in $1897$ documents. We further
sample up to 100 documents from each project country, to undersample popular
countries, resulting in 1290 documents, which we then sample randomly to get
the final 500 document sample.

####  4.1.2. Pre-annotation using baseline models

Labelling task is performed with Label Studio app888https://labelstud.io/.
With it, is possible to upload pre-annotated documents, reducing the effort of
the user to a simple revision of the pre-annotations. In order to insure a
variety of locations in our dataset, we truncate the documents at 4000
characters, thus encouraging the annotators to label more distinct documents.
To further help annotators find the candidate toponyms, we considered pre-
annotations which may ease the annotation process, letting the user to focus
on difficult cases without the need to tag all the locations in the text. To
explore this, we run a small test experiment prior the full annotation process
involving 4 users and 25 pilot documents. Two users were asked to tag pre-
annotated documents, the other 2 – documents without pre-annotations.
Annotation agreement does not vary among users with pre-annotated documents
and users without pre-annotated documents (remaining at around 0.90), however
the speed of their work improves by 7% (and this was during an adjustment
period where some aspects of the process were unclear to the labelers). Thus,
we proceed to create candidate toponyms using baseline NER models.

Pre-annotations are provided by two off-the-shelf NER models: Spacy
_en_core_web_md_ 999https://spacy.io/models/en#en_core_web_md, a pre trained
pipeline for English that includes NER components; and the roBERTa _xlm-
roberta-base-wikiann-ner_ 101010https://huggingface.co/Davlan/xlm-roberta-
base-wikiann-ner, a multilingual roBERTa based NER model finetuned on 20
annotated Wikipedia datasets111111https://huggingface.co/datasets/wikiann. The
pre-annotations are the results of the union of the predictions of the 2
models. In particular, in case of multi-word entities, if the predictions of
the 2 models overlap, the union of words of the 2 models is used as pre-
annotation (for instance, model 1 predicts _”the Mediterranean”_ and model 2
_”Mediterranean sea”_ , the resulting union is _”the Mediterranean sea”_).

####  4.1.3. Annotation schema

Defining clearly the meaning of location is crucial for the annotation task.
As shown in many other works, a location-related term could take different
meaning depending on the context (Sekine et al., 2002; Leidner, 2007). We
follow the main distinction by Gritta et al. (Gritta et al., 2019) between
_literal_ and _associative_ toponyms. A _literal_ toponym refers directly to a
physical location and an _associative_ toponym is only associated with a
place. For example, “Syria” is a literal toponym in “latest events in central
Syria”, but an associative in “Syria Red Cross aided border regions”. The
annotation task, then, comprises of reading the text, and annotating candidate
geotags, as well as creating new ones, and labeling them as either of these 2
categories: literal or associative. Instructions also include rules on
handling lists of locations, which are split in post-processing, as well as
how to annotate humanitarian-specific text such as names of reports,
organizations, and other adjectives that may include toponyms. The
instructions are available at shorturl.at/cxJN1.

####  4.1.4. Annotated geotagging dataset

A total of four 469 documents were annotated by annotators associated with the
DEEP platform, specializing in humanitarian data analysis. The median number
of toponyms per document is $25$, and the total number of toponyms annotated
is $11\,025$. Recall that each document has been truncated at 4000 characters
to allow a broader inclusion of different documents. The most common toponyms
in the data were _Libya, Syria, Tripoli, Afghanistan, Yemen, Niger, Sudan,
Venezuela, Nigeria, Somalia_. The annotated dataset includes _lead_id_ of the
document, _source_ url link, _text_ of the document (first 4000 characters)
and _annotations_ and is available at
https://github.com/embelliardo/HumSet_geolocation_annotations.

###  4.2. Annotation: geocoding

The second annotated dataset concerns the second task of this study: the
association of toponyms to a disambiguated location, including the GPS
coordinates. For this purpose, we have created a dataset of toponyms in
context mapped to an ID of GeoNames entry. GeoNames is a geographical database
that contains over 25 million geographical names, of which 4.8 million are
populated places, which also come with 13 million alternate names in local
alphabets121212https://www.geonames.org/about.html.

The toponyms were selected from a random selection of documents annotated in
the previous section. To prepare the toponyms for annotation, regular
expressions were used to split the lists of toponyms into individual locations
(i.e. “Al Hudaydah and Taizz governorates” to “Al Hudaydah governorate” and
“Taizz governorate”), associate noun modifiers to each location (i.e. “North
and South Italy” to “North Italy” and “South Italy”), rephrase possessive
pronouns (i.e. “City of New York’ to “New York City”), standardize special
characters, and remove dashes and apostrophes. These cleaned toponyms were
then matched with the database of GeoNames fields of _name_ and _alternate
name_ (which provides additional ways of writing the location name), which
were similarly preprocessed by standardizing special characters, and removing
dashes and apostrophes. The match was done in two ways. First match was
performed using longest consecutive word matching. The matches were then
filtered through a custom list which includes cardinals alone, common words,
and generic location signifiers. Second match was performed on the same
GeoNames fields using the Whoosh search engine that uses the Okapi BM25F
ranking function to retrieve
strings131313https://whoosh.readthedocs.io/en/latest/intro.html. This way of
matching allowed for more flexible positioning of keywords (i.e. “Tripoli
Airport” can be matched to “Tripoli International Airport”, which consecutive
word matching would miss). The search engine matches were then filtered based
on the “feature code” of the GeoNames entity, which describes what kind of
location it is. We keep locations having this code among the following (hand-
crafted based on relevance): administrative division (AD), populated place
(PPL), mountain (MT), sea (SEA), lake (LK), island (ISL) and airport (AIR).
Note that we use the same pipeline for the preparation of the toponyms for
geocoding using our automated algorithm below.

The annotation was performed using a custom-built tool that displays the
toponym, the document as its context, and a list of candidate GeoNames
entities, which are located on a map. The annotators were instructed to select
a GeoNames entity most likely to be referenced by the toponym, and if the
correct one cannot be found, to select “none”. The main author of this paper
performed the annotation, resulting in 561 unique document/toponym match pairs
from 39 documents, with 474 having non-empty matches, spanning 78 countries.
The annotated dataset includes _lead_id_ of the document, _toponym_ , _match_
and the _GeoNames ID_ of the correct match, and is available at
https://github.com/embelliardo/HumSet_geolocation_annotations.

##  5\. Customizing Geo-Location for Humanitarian Texts

In the next sections, we describe the evaluation of state-of-the-art
geotagging methods and tuning of these methods with the annotated data.

###  5.1. Geotagging

Using the annotated geotagging dataset, we were now able to evaluate the off-
the-shelf NER models – Spacy and roBERTa – used in pre-selection of the
toponyms. We compute two versions of the metrics: exact agreement considers
only exact string matches to the ground-truth labels (subscripts $e$ in Table
2), whereas partial agreement considers any substring overlap (subscript $p$).
Note that partial scoring includes exact matches. We define agreement between
the output of the model and the ground-truth as the number of toponyms
predicted by the model which are in the annotated set, divided by the union of
toponyms of the two sets. We further use the standard definitions of precision
and recall. In the case of multiple correct matches, we count such matches as
one (by weighting each match proportional to the number of sub-matches), such
that we do not inflate the number of matched toponyms. Table 2 reports the
performance of the baseline models (as well as the tuned ones, described
below). Both algorithms were evaluated on the entire annotated set using
10-fold cross-validation. The cross-validation is run in two ways: without
stratification, and with stratification by the country of the project, such
that the training data does not have information about the test project
country, making it a more difficult task.

Table 2. Agreement, precision, recall, and F1 for geotagging methods: mean (standard deviation) over 10 folds. Exact matches with subscript $e$ and partial with subscript $p$. Tested with and without stratification (“ST”) by country of the project. Method | Ae | Pe | Re | F1e | Ap | Pp | Rp | F1p
---|---|---|---|---|---|---|---|---
roBERTabaseline | .63(.03) | .73(.02) | .72(.03) | .72(.02) | .74(.03) | .85(.02) | .85(.03) | .85(.02)
roBERTatuned | .75(.03) | .78(.02) | .81(.02) | .79(.02) | .87(.03) | .92(.02) | .95(.01) | .93(.01)
roBERTatuned_ST | .72(.03) | .79(.04) | .76(.03) | .78(.03) | .85(.04) | .94(.02) | .90(.04) | .92(.02)
Spacybaseline | .53(.08) | .74(.07) | .57(.08) | .64(.07) | .64(.07) | .90(.04) | .68(.08) | .78(.05)
Spacytuned | .74(.03) | .79(.03) | .80(.03) | .79(.02) | .85(.02) | .92(.03) | .92(.03) | .91(.01)
Spacytuned_ST | .65(.06) | .77(.06) | .70(.07) | .74(.05) | .77(.06) | .92(.04) | .82(.07) | .87(.04)
Combined  | .72(.03) | .74(.04) | .80(.02) | .78(.03) | .87(.03) | .89(.03) | .97(.00) | .93(.02)
Combined_ST | .70(.05) | .73(.05) | .77(.03) | .75(.04) | .86(.04) | .90(.04) | .95(.01) | .92(.03)

For both algorithms, and both evaluation methods, we find the tuning of the
model with additional data to improve performance, both in terms of precision
and recall. For roBERTa, the improvement in F1 is from 0.72 to 0.79 (not
stratified) for exact matches and from 0.85 to 0.93 (not stratified) for
partial ones. The improvement is even more for Spacy, a 15 percentage point
improvement in F1 to 0.79 (not stratified) for exact matches, and a 13 point
improvement in F1 to 0.91 (not stratified) for partial. Interestingly, the
variance of performance of the Spacy model decreases with additional tuning,
however in the stratified testing it remains higher. As expected, the
performance is lower when the data is stratified by project country, but there
are still some improvements. We also combine the two models by taking a union
of the matches produced by both algorithms (and a largest span of those that
partially overlap). This approach favors the recall, achieving 0.97 in partial
matching, while still having a precision of 0.89. The training brings the
performance of the two algorithms closer (with Spacy having largest
improvement), but other considerations may be important in the deployment of
the models. Whereas Spacy model takes about 6.5MB, roBERTa is much larger at
1.1GB. Similarly, Spacy runs faster, about 3.2 times faster than roBERTa. To
conclude, if one favors recall, a combined model should be used; if a low
variance in performance is desired, roBERTa may be a better choice; but if a
lightweight model is needed for fast processing, Spacy may be a better
selection.

Figure 1 shows the improvement in performance as the amount of training data
is increased. We find a rise in performance within the first 10% (about 50
documents), which gradually increases especially for the partial matches,
which is true for both models, and for both testing using perfect match and
partial match. roBERTa benefits especially from the new data, achieving F1
score above 0.9 with additional data (for partial scoring).

Figure 1. Average $F_{1}$ score and standard deviation on 10 folds cross-
validation at different sample size. $F_{1}$ score is computed for Spacy and
roBERTa models, baseline and finetuned, and for perfect and partial matches.

[F1 score increases with sample size, with rise in performance within the
first 10%]F1 score increases with sample size, with rise in performance within
the first 10%. roBERTa - partial and Spacy - partial significantly benefit
from the training with full sample size

Next, we examine the disagreements in the extracted toponyms by the different
algorithms. In Table 3 we show several examples of a text fragment and the
associated toponyms extracted by the two algorithms: in their baseline form
and finetuned. The first example talks about a boat (“VOS vessel”) named
Theia, which is extracted as a location by the roBERTa baseline model, but is
ignored by the finetuned one (Spacy performs correctly even as a baseline).
The second example has the phrase “Syrian Government”, which is an associative
toponym that we do not select for the present task. The tuned roBERTa model
learn to ignore such cases. Third example is similar, but we find that the
training benefits Spacy, such that the associative toponym is not selected. In
the fourth example the location string is mixed with numeric characters.
Whereas the baseline roBERTa model is not able to identify it, Spacy
identifies the word, but with additional numerical characters. Training
improves roBERTa, but makes Spacy ignore the string (note that a combined
model would still find the toponym). The fifth example shows how the trained
model (in this case roBERTa) is capable of recognising long lists of noun
modifiers associated with “districts”. Finally, the last example shows how the
trained models are able to identify location at a very granular level, in this
case an hospital building. Although not specifically focusing on particular
kinds of toponyms, we find that smaller locations such as hospitals, airports,
refugee camps, and others may especially necessitate training data to be
caught by the named entity extractors.

Table 3. Sample of toponyms with their context predicted by baseline and finetuned models. Correct annotation in bold. roBERTatuned |  roBERTabaseline |  Spacytuned |  Spacybaseline
---|---|---|---
_1\. two VOS vessels. Two will be located on board the VOS Theia, two will be
located at Aden port, and one will …_
None ✓ |  VOS Theia |  None ✓ |  None ✓
_2\. … and volatile in June, with tensions between the Syrian Government and
‘reconciled’ non-state armed groups reported …_
None✓ |  Syrian Government |  None ✓ |  None ✓
_3\. The US Government Congratulates Buhari in Spite of Violent and Corrupt
Election_
None✓ |  None✓ |  None✓ |  US
_4\. Plateau(5), Taraba(3), Gombe(1), Kaduna(1), Kwara(1), FCT(1), Benue(2),
Rivers(1) Kogi(1)  …_
Kwara✓ |  None |  None |  Kwara(1
_5\. However, clashes intensified in At Tuhayat and Zabid districts of
Hudaydah city … _
At Tuhayat and Zabid districts ✓ |  At Tuhayat |  Tuhayat |  None
_6\. sources report 17 dead and eight wounded, currently in treatment at Am-
Timan hospital _
Am-Timan hospital✓ |  None |  Am-Timan hospital✓ |  None

###  5.2. Geocoding

####  5.2.1. Baseline Approaches

Considering the geocoding task – the mapping of toponyms to a unique set of
geo-coordinates, and potentially to other meta-data associated with a location
– the literature is sparse in this area. We consider two approaches to
geocoding from the previous literature. Buscaldi & Magnini (Buscaldi and
Magnini, 2010) propose an approach based on an iterative resolution of
references by favoring “unambiguous” ones, with an assumption that the
references appearing at greater frequency (such as countries) are less
ambiguous. Intuitively, for each candidate geolocation for a toponym, the
algorithm considers resolved references in the context around the candidate,
and computes inverse distance to the unambiguous geolocations. The candidate
geolocation having the shortest distance to the unambiguous known references
in the text is chosen as the best. Following the findings of the paper, we do
not implement the text distance between the toponym and references in the
text. We considered as unambiguous toponyms which had only one GeoNames match
and had to be in one of the countries mentioned in the document, as well as
the countries themselves. The second approach by Chen, Vasardani, and Winter
(Chen et al., 2018) also uses the contextual information around a toponym. It
involves the clustering of the candidate geolocations for all toponyms found
in a document, finding the best clustering radius, and choosing the candidate
geolocations from the biggest cluster. The model assumes that the toponyms
found in a document are located near each other. It further does not provide a
tie-breaking mechanism for choosing the best geolocation within the selected
cluster (so we break ties randomly).

####  5.2.2. FeatureRank

As an alternative to the general-purpose geocoding algorithms, in this work we
propose a custom feature-based geocoding method tailored for the humanitarian
domain. This method, FeatureRank, considers geopolitical and population
features of candidate locations extracted from GeoNames, as well as a
document-wide distribution of the candidate locations for all toponyms, thus
capturing both local and global knowledge.

First, the toponyms are preprocessed as in the Section 4.2. Recall that the
preprocessing involves several steps in cleaning and splitting toponyms,
searching GeoNames database using string matching and search engine, and
filtering the candidate geolocation set using a set of codes. This results in
a set of candidate geolocations for each toponym in a document. A variant of
our algorithm SearchFeatureRank also uses a search engine to query for
candidate locations (instead of substring match). Similar to the baseline
models, we summarize the distribution of the geolocation candidates in the
documents by computing a document-wide country distribution. To compute this
distribution, for each toponym in the document we first create a country
distribution of its geolocation candidates, and finally average them for a
document-wide distribution. Note that we are assuming the document is largely
concerning one country, which we find to be the case for most of the documents
in our dataset, but is not necessarily true for other settings. We then
geocode each toponym.

To do this, we compute a set of features for each geolocation candidate for a
toponym:

  * •

IsCapital: it is 1 if the GeoNames feature code is PPLC, and only if the
candidate comes from the string match.

  * •

IsCountry: it is 1 if the GeoNames feature code is PCLI or PCLS, and only if
the candidate comes from the string match.

  * •

AdminLevel: the numeric level (1-5) of the administrative unit (ADM or PPL) in
the GeoNames feature code; 0 for capital cities or countries; 6 for
geolocations without a level.

  * •

IsCity: it is 1 if the GeoNames feature code begins with PPL (some populated
location).

  * •

Population: population provided by GeoNames.

  * •

DocCountryMatch: the probability of the country of the candidate geolocation
in the document country distribution.

Finally, the candidates are ranked using the following order: IsCapital
descending, IsCountry descending, DocCoun- tryMatch descending, AdminLevel
ascending, IsCity descending, Population descending. The algorithm selects the
best results according to the ranking strategy, therefore a candidate is
always chosen even if none of the candidates matches the correct location. A
threshold rule has proven effective to filter out unlikely solutions: if
AdminLevel $>$ 5 and the candidate Country is not the country with highest
DocCountryMatch, the location is discarded.

####  5.2.3. Evaluation

We evaluate these algorithms on the Geocoding dataset described above. In
particular, we consider several metrics:

  * •

For correct prediction, we consider two cases:

    * –

True Positive: match exists and correctly predicted

    * –

True Negative: match does not exist and is correctly ignored

    * –

Correct: sum of the two above

  * •

For incorrect prediction, we consider three cases:

    * –

False Positive (new selection): match does not exist, but some value is
wrongly predicted

    * –

False Negative: match exists but is not selected (i.e excluded by threshold)

    * –

False Positive (wrong selection): match exists but wrong candidate is selected

    * –

Incorrect: sum of the three above

  * •

For False Positive (wrong selection) we compute the average distance of the
incorrect guess to the correct one.

  * •

For the toponyms which were tagged as non-locations, how many were identified
as locations.

We can see from Table 4 that an approach based on features is better suited
than other methods based on geographical distances. Both FeatureRank and
SearchFeatureRank reach similar performances, with correct prediction in 79%
and 80% of the cases, respectively. The only notable difference is in the
median distance error, which decreases in the search engine version. Note that
during the geocoding annotation process, it was possible to indicate whether a
toponym is correct, and in 18 cases we found that the toponym was selected
incorrectly during the first round of annotations. Using this information, we
are able to test the performance of our models on such cases, and we find that
FeatureRank correctly does not match these cases to anything 50% of the time
(compared to 0% by BM and 22% by Chen baselines).

Table 4. proportion of correct and incorrect matches, and median distance from prediction to ground truth for wrong matches for geocoding methods. We propose FeatureRank and SearchFeatureRank.  | BM | Chen | FeatureRank | SearchFeatureRank
---|---|---|---|---
Locations (count) | 543 | 543 | 543 | 543
True Positive | 0.58 | 0.21 | 0.71 | 0.72
True Negative | 0.00 | 0.01 | 0.08 | 0.08
Correct | 0.58 | 0.22 | 0.79 | 0.80
False Positive (new selection) | 0.13 | 0.12 | 0.05 | 0.05
False Negative | 0.00 | 0.03 | 0.04 | 0.03
False Positive (wrong selection) | 0.29 | 0.63 | 0.12 | 0.12
Incorrect | 0.42 | 0.78 | 0.21 | 0.20
Median dist. (Km) | 219.36 | 2610.94 | 83.11 | 61.56
Non-locations (count) | 18 | 18 | 18 | 18
Correct | 0.00 | 0.22 | 0.50 | 0.44

When we consider the 29 locations known to be in US & Europe separately from
all others, we find that our algorithm attains 100% accuracy for locations in
US & Europe and 84% for those outside (and for those toponyms which are
locations, but which are not in GeoNames, the algorithm performs the worst, at
48% accuracy). However, in terms of guessed locations, only 69% of the
locations our algorithm guesses to be in US & Europe are correct, compared to
80% of those guessed in other locales. In particular, our algorithm works well
for locations in _Libya_ , _Colombia_ , _Afghanistan_ and _Ecuador_ , where
100% of locations are correctly predicted. We note that the worst performing
countries are _Panama_ , _Mexico_ and _Paraguay_ , for which the algorithm
never predicts correctly (however, we also do not have any projects about
_Panama_ and _Paraguay_ in the training data, though locations from those
countries could be mentioned in other projects). We hypothesize it is due to
the similar naming and language conventions among these countries.

##  6\. Application study

Finally, we apply the tuned toponym extraction and the custom FeatureRank
geocoding algorithm to the HumSet dataset. Although we do not have the ground
truth for these labels, we examine potential biases in the locations that the
baseline models find, compared to our tuned ones. In total, we annotate 6733
documents, extracting $13\,967$ distinct locations. Figure 2 shows the number
of locations identified by the two baseline models, and two tuned models, by
country (left) and by country grouped by their human development index
(right). We find that, generally, the number of matched locations increases
for both models. However, for roBERTa, the matches in US decrease
dramatically, which is also reflected in the lower matches in countries with
very high HDI. Note that this effect was not intended by the tuning of the
algorithm, and is an interesting side effect. We can also notice that the
number of locations detected increases even in countries that were not in the
training set, like Democratic Republic Of the Congo (CD) and Mozambique (MZ).

Figure 2. Number of locations detected in HumSet by baseline Spacy (s) and
roBERTa (r) models, as well as the tuned ones by country (left, top 15
countries shown), and by the country’s human development index (right).

[Overall, roBERTa trained model finds fewer locations in US and countries with
very-high HDI]Overall, roBERTa trained model finds fewer locations in US and
countries with very-high HDI. On the other hand, it find more locations in
countries from low to high HDI. Spacy increases the number of locations found
across all the HDI levels.

Figure 3 shows a map with areas highlighted in blue where the baseline roBERTa
model finds more locations, and highlighted in red where the tuned roBERTa
model finds more. We observe that the baseline model favors United States and
Europe, and Australia – countries with high and very high development index.
In fact, the tuned model predicts 55% fewer matches in US. On the other hand,
the countries highlighted in yellow – those for which there are projects in
the HumSet data – show an increase in matches, especially Trinidad and Tobago
has more than twice as many matches (from 92 by baseline to 191 by tuned),
Domenican Republic increases by 17%, Bahamas by 16%, Sudan 14%, and Lybia by
12%. However, even countries which do not have projects in the HumSet
documents have more matches, including Morocco at 102% more, South Korea 90%,
North Korea 81%, and Myanmar 64%. As we do not know the ground truth of these
matches, more work needs to be done to understand whether this is due to
increase in accuracy, or whether this is an evidence of a new geographic bias.
If these new matches are indeed correct, this would point to a
generalizability of these models beyond the countries of the tuned data.

Figure 3. Map of the difference in locations identified by roBERTa: in red
more by the tuned model, and in blue more by the baseline model. Countries in
yellow are those for which HumSet contains projects.

[Baseline models find more locations in Western countries. Finetuned models
finds more locations in countries included in HumSet]Baseline models find more
locations in Western countries, in particular US, European countries and
Australia. Finetuned models finds more locations in countries included in
HumSet

##  7\. Discussion

As we attempted to improve the performance of state-of-the-art NER for
geolocation of entities in humanitarian texts, we discovered that training
data from the humanitarian domain has not only improved accuracy of the tools,
but also possibly helped alleviate the geographic bias of these tools toward
the Western countries. Our findings during the application of the baseline and
tuned algorithm to the HumSet data suggest that data extraction tools should
be tested for bias in a systematic way, as proposed by algorithmic auditing
proponents (Vecchione et al., 2021). The social justice concerns auditors of
economic and governmental systems extend to the use of AI tools in regards to
possibly vulnerable populations, such as those needing humanitarian
assistance. Similar concerns have been raised in the use of AI for medical
diagnosis (Seyyed-Kalantari et al., 2021), hiring (Raghavan et al., 2020), and
poverty mapping (Beiró and Kalimeri, 2022; Sartirano et al., 2023). Auditing
is especially necessary for black-box algorithms wherein coverage and
completeness are impossible to ascertain, unlike for algorithms that use
gazetteers or databases.

In humanitarian context, geographic accuracy is imperative for the accurate
assessment of needs and their development over time. During the development of
the annotated resources, we encountered several peculiarities of geolocations
present in humanitarian texts. First, we often encounter geo-relevant
organization names, such as “Syrian Arab Red Crescent” or “European
Commissioner for Crisis Management” that indicate their affiliation or source,
but not necessarily the location of their current involvement. We observed
that this often would decrease the performance of distance-based algorithms
that average locations of found entity candidates. A specialized module to
handle such information may benefit the mapping of the entities involved in a
project. Second, locations often have qualifiers such as “western Yemen” or
“north of Baghdad”, necessitating for additional semantics in location
representation that could capture both the directionality and uncertainty of
the location. Third, rapid developments on the ground, such as the building or
dismantling refugee camps, should be added to the geographic resources used
for humanitarian aid on an ongoing basis.

More research is needed to improve these tools and evaluate them on a larger
set of documents. Further, thus far only documents in English language were
considered (although many humanitarian organizations publish their reports in
English for international consumption). This paper does not use any personal
data, instead it uses only published reports and news articles, available
under the Apache 2.0 license. We hope the annotated resources and guidelines
provided in this paper will spur further work on tuning NER tools to handle
data around vulnerable populations.

###### Acknowledgements.

The authors gratefully acknowledge the support from the Lagrange Project of
the Institute for Scientific Interchange Foundation (ISI Foundation) funded by
Fondazione Cassa di Risparmio di Torino (Fondazione CRT).

## References

  * (1)
  * American Red Cross (2022) American Red Cross. 2022\.  What is the Red Cross Global Network?  https://www.redcross.org/about-us/news-and-events/news/2022/what-is-the-red-cross-global-network.html.  September 16, 2022..
  * Beiró and Kalimeri (2022) Mariano G Beiró and Kyriaki Kalimeri. 2022.  Fairness in vulnerable attribute prediction on social media.  _Data Mining and Knowledge Discovery_ 36, 6 (2022), 2194–2213.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ 26 (2013).
  * Buscaldi and Magnini (2010) Davide Buscaldi and Bernardo Magnini. 2010.  Grounding toponyms in an italian local news corpus. In _Proceedings of the 6th workshop on geographic information retrieval_. 1–5.
  * Chen et al. (2018) Hao Chen, Maria Vasardani, and Stephan Winter. 2018\.  Disambiguating fine-grained place names from descriptions by clustering.  _arXiv preprint arXiv:1808.05946_ (2018).
  * Collobert et al. (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011\.  Natural language processing (almost) from scratch.  _Journal of machine learning research_ 12, ARTICLE (2011), 2493–2537.
  * Dalton et al. (2016) Craig M Dalton, Linnet Taylor, and Jim Thatcher. 2016\.  Critical data studies: A dialog on data and space.  _Big Data & Society_ 3, 1 (2016), 2053951716648346\.
  * Dernoncourt et al. (2017) Franck Dernoncourt, Ji Young Lee, and Peter Szolovits. 2017\.  NeuroNER: an easy-to-use program for named-entity recognition based on neural networks. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. 97–102.
  * Fekih et al. (2022) Selim Fekih, Nicolo’ Tamagnone, Benjamin Minixhofer, Ranjan Shrestha, Ximena Contla, Ewan Oglethorpe, and Navid Rekabsaz. 2022\.  HumSet: Dataset of Multilingual Information Extraction and Classification for Humanitarian Crises Response. In _Findings of the Association for Computational Linguistics: EMNLP 2022_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 4379–4389.  https://aclanthology.org/2022.findings-emnlp.321
  * Graham (2015) Mark Graham. 2015\.  The hidden biases of Geodata.  https://www.theguardian.com/news/datablog/2015/apr/28/the-hidden-biases-of-geodata.  The Guardian. [Accessed April 10, 2023].
  * Graham and De Sabbata (2015) Mark Graham and Stefano De Sabbata. 2015.  Mapping information wealth and poverty: the geography of gazetteers.  _Environment and Planning A_ 47, 6 (2015), 1254–1264.
  * Graham et al. (2014) Mark Graham, Scott A Hale, and Devin Gaffney. 2014\.  Where in the world are you? Geolocation and language identification in Twitter.  _The Professional Geographer_ 66, 4 (2014), 568–578.
  * Gritta et al. (2019) Milan Gritta, Mohammad Taher Pilevar, and Nigel Collier. 2019.  A pragmatic guide to geoparsing evaluation: Toponyms, Named Entity Recognition and pragmatics.  _Language Resources and Evaluation_ 54 (09 2019).  https://doi.org/10.1007/s10579-019-09475-3
  * Imran et al. (2015) Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015.  Processing social media messages in mass emergency: A survey.  _ACM Computing Surveys (CSUR)_ 47, 4 (2015), 1–38.
  * Imran et al. (2022) Muhammad Imran, Umair Qazi, and Ferda Ofli. 2022.  Tbcov: two billion multilingual covid-19 tweets with sentiment, entity, geo, and gender labels.  _Data_ 7, 1 (2022), 8.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of naacL-HLT_ , Vol. 1. 2.
  * Lai et al. (2022) Kelvin Lai, Jeremy R Porter, Mike Amodeo, David Miller, Michael Marston, and Saman Armal. 2022\.  A natural language processing approach to understanding context in the extraction and geocoding of historical floods, storms, and adaptation measures.  _Information Processing & Management_ 59, 1 (2022), 102735\.
  * Leidner (2007) Jochen L. Leidner. 2007\.  Toponym resolution in text: Annotation, evaluation and applications of spatial grounding.  _SIGIR Forum_ 41 (01 2007), 124–126.
  * Liang et al. (2020) Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020.  Bond: Bert-assisted open-domain named entity recognition with distant supervision. In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_. 1054–1064.
  * Logeswaran et al. (2019) Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019.  Zero-Shot Entity Linking by Reading Entity Descriptions. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 3449–3460.
  * Malik et al. (2015) Momin Malik, Hemank Lamba, Constantine Nakos, and Jürgen Pfeffer. 2015.  Population bias in geotagged tweets. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 9. 18–27.
  * Middleton et al. (2018) Stuart E Middleton, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018\.  Location extraction from social media: Geoparsing, location disambiguation, and geotagging.  _ACM Transactions on Information Systems (TOIS)_ 36, 4 (2018), 1–27.
  * Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007.  A survey of named entity recognition and classification.  _Lingvisticae Investigationes_ 30, 1 (2007), 3–26.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. 701–710.
  * Poletto et al. (2021) Fabio Poletto, Yunbai Zhang, André Panisson, Yelena Mejova, Daniela Paolotti, and Sylvain Ponserre. 2021.  Developing Annotated Resources for Internal Displacement Monitoring. In _Companion Proceedings of the Web Conference 2021_. 136–144.
  * Radke et al. (2018) Mansi A Radke, Nitin Gautam, Akhil Tambi, Umesh A Deshpande, and Zareen Syed. 2018.  Geotagging Text Data on the Web—A Geometrical Approach.  _IEEE Access_ 6 (2018), 30086–30099.
  * Raghavan et al. (2020) Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020.  Mitigating bias in algorithmic hiring: Evaluating claims and practices. In _Proceedings of the 2020 conference on fairness, accountability, and transparency_. 469–481.
  * Rama et al. (2020) Daniele Rama, Yelena Mejova, Michele Tizzoni, Kyriaki Kalimeri, and Ingmar Weber. 2020\.  Facebook ads as a demographic tool to measure the urban-rural divide. In _Proceedings of The Web Conference 2020_. 327–338.
  * Sartirano et al. (2023) Daniele Sartirano, Kyriaki Kalimeri, Ciro Cattuto, Enrique Delamónica, Manuel Garcia-Herranz, Anthony Mockler, Daniela Paolotti, and Rossano Schifanella. 2023.  Strengths and limitations of relative wealth indices derived from big data in Indonesia.  _Frontiers in big Data_ 6 (2023), 10.
  * Sekine et al. (2002) Satoshi Sekine, Kiyoshi Sudo, and Chikashi Nobata. 2002\.  Extended Named Entity Hierarchy. In _Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02)_. European Language Resources Association (ELRA), Las Palmas, Canary Islands - Spain.  http://www.lrec-conf.org/proceedings/lrec2002/pdf/120.pdf
  * Sevgili et al. (2022) Özge Sevgili, Artem Shelmanov, Mikhail Arkhipov, Alexander Panchenko, and Chris Biemann. 2022\.  Neural entity linking: A survey of models based on deep learning.  _Semantic Web_ Preprint (2022), 1–44.
  * Seyyed-Kalantari et al. (2021) Laleh Seyyed-Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. 2021\.  Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations.  _Nature medicine_ 27, 12 (2021), 2176–2182.
  * Singh et al. (2019) Jyoti Prakash Singh, Yogesh K Dwivedi, Nripendra P Rana, Abhinav Kumar, and Kawaljeet Kaur Kapoor. 2019.  Event classification and location prediction from tweets during disasters.  _Annals of Operations Research_ 283 (2019), 737–757.
  * Statista (2022) Statista. 2022\.  Leading countries based on number of Twitter users as of January 2022.  https://www.statista.com/statistics/242606/number-of-active-twitter-users-in-selected-countries/.  [Accessed April 10, 2023].
  * Vecchione et al. (2021) Briana Vecchione, Karen Levy, and Solon Barocas. 2021\.  Algorithmic auditing and social justice: Lessons from the history of audit studies.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–9.
  * Wikipedia (2022) Wikipedia. 2022\.  List of Wikipedias.  https://meta.wikimedia.org/wiki/List_of_Wikipedias.  [Accessed April 10, 2023].
  * Wu et al. (2020) Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020.  Scalable Zero-shot Entity Linking with Dense Entity Retrieval. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 6397–6407.
  * Yadav and Bethard (2018) Vikas Yadav and Steven Bethard. 2018.  A Survey on Recent Advances in Named Entity Recognition from Deep Learning models. In _Proceedings of the 27th International Conference on Computational Linguistics_. 2145–2158.
  * Zheng et al. (2018) Xin Zheng, Jialong Han, and Aixin Sun. 2018.  A survey of location prediction on twitter.  _IEEE Transactions on Knowledge and Data Engineering_ 30, 9 (2018), 1652–1671.
  * Zou et al. (2018) Lei Zou, Nina SN Lam, Heng Cai, and Yi Qiang. 2018\.  Mining Twitter data for improved understanding of disaster resilience.  _Annals of the American Association of Geographers_ 108, 5 (2018), 1422–1441.