# UrbanKGent: A Unified Large Language Model Agent Framework for Urban
Knowledge Graph Construction

Yansong Ning  AI Thrust, The Hong Kong University of Science and Technology
(Guangzhou<EMAIL_ADDRESS>and  Hao Liu  AI Thrust, The Hong
Kong University of Science and Technology (Guangzhou)CSE, The Hong Kong
University of Science and Technology<EMAIL_ADDRESS>
(2018)

###### Abstract.

Urban knowledge graph has recently worked as an emerging building block to
distill critical knowledge from multi-sourced urban data for diverse urban
application scenarios. Despite its promising benefits, urban knowledge graph
construction (UrbanKGC) still heavily relies on manual effort, hindering its
potential advancement. This paper presents UrbanKGent, a unified large
language model agent framework, for urban knowledge graph construction.
Specifically, we first construct the knowledgeable instruction set for
UrbanKGC tasks (such as relational triplet extraction and knowledge graph
completion) via heterogeneity-aware and geospatial-infused instruction
generation. Moreover, we propose a tool-augmented iterative trajectory
refinement module to enhance and refine the trajectories distilled from GPT-4.
Through hybrid instruction fine-tuning with augmented trajectories on
Llama-2-13B, we obtain the UrbanKGC agent, UrbanKGent-13B. We perform a
comprehensive evaluation on two real-world datasets using both human and GPT-4
self-evaluation. The experimental results demonstrate that UrbanKGent-13B not
only can significantly outperform 21 baselines in UrbanKGC tasks, but also
surpass the state-of-the-art LLM, GPT-4, by more than 10% with approximately
20 times lower cost. We deploy UrbanKGent-13B to provide online services,
which can construct an UrbanKG with thousands of times richer relationships
using only one-fifth of the data compared with the existing benchmark. Our
data, code, and opensource UrbanKGC agent are available at
https://github.com/usail-hkust/UrbanKGent.  Figure 1. The framework overview
of UrbanKGent.

urban knowledge graph, knowledge graph construction, large language model
agent

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Artificial
intelligence; Natural language processing; Information extraction

##  1\. Introduction

Urban Knowledge Graph (UrbanKG) aims to model intricate relationships and
semantics within a city by extracting and organizing urban entities (e.g.,
POIs, road networks, etc.) into a multi-relational heterogeneous graph. As an
emerging building block, multi-sourced urban data are widely used to construct
an UrbanKG to provide critical knowledge for various knowledge-enhanced urban
downstream tasks, such as traffic management, pollution monitoring, and
emergency response (Sun et al., 2017; Chen et al., 2016; Cheng et al., 2018).
UrbanKG has gradually become an essential tool of the modern smart city.

In prior literature, many efforts have been devoted to urban knowledge graph
construction (UrbanKGC) using massive urban data sources. In particular, one
commonly used approach (Wang et al., 2021; Liu et al., 2022; Ning et al.,
2023) is to extract entities from structured urban data (e.g., geographic
data, city sensor data, and traffic data) and define the relationships between
obtained urban entities based on manually designed rules or patterns. However,
these approaches suffer heavy reliance on a deep understanding of the
application domain and are labor-intensive. Recently, inspired by the success
of the Large Language Models (LLMs) in knowledge graph construction (Xie et
al., 2023; Zheng et al., 2021; Yao et al., 2023), the LLMs have been adopted
to improve UrbanKGC. For instance, GeoLM (Wang et al., 2023b) is proposed for
spatial grounded entity recognition and relation extraction. K2 (Deng et al.,
2023) retrains the Llama-2-7B model on a huge annotated geoscience text corpus
for geospatial relation extraction. Nevertheless, these works rely on
_annotated corpus_ and _model retraining_ , which may discourage researchers
from adopting it for their own work and thus limit the application of UrbanKG.

LLM agent (et al, 2023a, b) has recently emerged and shown remarkable zero-
shot capability for autonomous domain-specific task completion. For example,
Voyager (Wang et al., 2023c) is a LLM-powered agent for zero-shot game
exploration without re-training, and LLMLight (Gur et al., 2023) is a traffic
signal control agent with zero-shot LLM reasoning ability. These studies
motivate us to construct tailored LLM agents to address the aforementioned
limitations in UrbanKG construction.

Figure 2. Illustrative example of urban relational triplet extraction and
knowledge graph completion. (a) The heterogeneous relationship understanding
limitation of LLMs can be addressed by injecting prior urban knowledge into
instruction. (b) The geospatial computing limitation of LLMs can be alleviated
by invoking external geospatial tools.

In fact, constructing an LLM agent compatible with various UrbanKGC tasks is a
non-trivial problem due to the following two challenges: _(1) Challenges 1:
How to adapt LLMs for UrbanKGC?_ LLMs may not align well with the specific
task due to the gap (Fu et al., 2023b) between the natural language processing
corpus for training LLMs and the domain-specific corpus in urban domain (Lin
et al., 2023). For example, the urban text data is usually heterogeneous and
contains multifaceted urban knowledge (e.g., spatial, temporal, and functional
aspects) (Deng et al., 2023). As shown in Figure 2(a), the text description of
_”Columbia University”_ reflects its geographic spatial locations (i.e.,
spatial relationship), construction timelines (i.e., temporal relationship),
and how it provides educational service for the city (i.e., functional
relationship). LLMs may require a tailored alignment to understand
heterogeneous urban relationships to extract these urban spatial, temporal,
and functional relations accurately. _(2) Challenges 2: How to improve the
capacity of LLMs for UrbanKGC?_ The effectiveness of LLMs for urban knowledge
graph construction is restricted by their feeble numerical computation
capacity (Frieder et al., 2023; Zhang et al., 2023b), leading to their
disability in complex geospatial relationship extraction (Bhandari et al.,
2023; Mooney et al., 2023). However, the urban geospatial relationship plays a
vital role in urban semantic modeling (Li et al., 2023c) and has been widely
incorporated in previous UrbanKGs (Ning et al., 2023; Liu et al., 2023). As
can be seen in Figure 2, extracting _”disconnected”_ relation between the geo-
entity _”Columbia University”_ and _”Empire State Building”_ is useful for
urban geo-semantic modeling. Accurately extracting such geospatial
relationship demands necessary geospatial computing (e.g., utilizing latitude
and longitude for distance calculation) and reasoning (i.e., deriving
calculation results for geospatial relation reasoning) capabilities. It is
appealing to improve the geospatial computing and reasoning ability (e.g.,
invoking external tools for calculation) of LLMs to satisfy the UrbanKGC task
requirement.

To address the aforementioned challenges, in this study, we propose
UrbanKGent, a unified LLM agent framework for automatic UrbanKG construction.
Figure 1 illustrates the overview of UrbanKGent. For a given city, we first
generate a knowledgeable instruction set for UrbanKGC tasks (e.g., relational
triplet extraction and knowledge graph completion) from urban geographic and
text data sources. By heterogeneity-aware and geospatial-infused instruction
generation, as shown in Figure 2(a), various urban spatiotemporal relationship
knowledge can be encoded into the instruction, which facilitates alignment
between LLMs with the target UrbanKGC tasks. Moreover, we propose a tool-
augmented iterative trajectory refinement module to enhance and refine the
trajectory derived by distilling GPT-4 with the above constructed
instructions. Based on geospatial tool augmentation and self-refinement, the
deficiency of LLMs in geospatial computing and reasoning could be alleviated,
and unfaithful trajectories could be filtered out. Finally, we perform hybrid
instruction fine-tuning based on the enhanced and refined trajectories on
Llama 2-7B and Llama 2-13B variants (Touvron et al., 2023) by using LoRA (Hu
et al., 2021). The obtained agent, UrbanKGent-13B, is feasible for completing
multiple UrbanKGC tasks in a cost-effective way (i.e., no extra GPT-API cost
needed).

We conduct comprehensive experiments on two UrbanKGC tasks in two metropolises
(New York City and Chicago) using both human evaluation and GPT-4-based self-
evaluation. The empirical results validate the effectiveness of the proposed
LLM agent for completing various UrbanKGC tasks. Moreover, we deploy the
constructed agent, UrbanKGent-13B, online for UrbanKGC service, which can
extract the same scale of triplets and entities of existing UrbanKG benchmark
(Ning et al., 2023) using only one-fifth of data, and even expand the types of
relations by thousands of times.

Our contributions are summarized as follows:

  * •

We propose the first UrbanKGC agent construction framework UrbanKGent, and
release the UrbanKGent-13B to provide real-world UrbanKGC
service111https://htmlpreview.github.io/?https://raw.githubusercontent.com/usail-
hkust/UrbanKGent/main/UrbanKGent%20Demo/index.html, which offers new
opportunities to advance UrbanKG studies.

  * •

We propose a knowledgeable instruction generation module and a tool-augmented
iterative trajectory refinement method, which align LLMs to UrbanKGC tasks and
compensate for their geospatial computing and reasoning inability.

  * •

Extensive experiments on two real-world datasets validate the effectiveness of
the proposed framework and uncover its exceptional performance across various
UrbanKGC tasks.

##  2\. UrbanKGC Data Description

This section introduces the process of data collection and preprocessing
details.

###  2.1. Data Collection

We first acquire urban knowledge for two large cities New York City and
Chicago from two data sources. Table 1 summarizes the statistics of the raw
datasets.

####  2.1.1. Geographic Data

The geographic data provides critical urban spatial structure information and
functional semantics, which has been widely used in previous UrbanKG studies
(Ning et al., 2023; Liu et al., 2023, 2021; Zhang et al., 2023a).

Area-Of-Interst (AOI) Data. AOI data describes the urban spatial area
structure, including urban commercial areas (e.g., shopping centers),
residential areas (e.g., communities and neighbors), and so on. In this work,
we first follow UUKG (Ning et al., 2023) to acquire the AOI name and geometry
value from NYC Gov222https://www.nyc.gov/ and CHI
Gov333https://www.chicago.gov/. Next, we use the AOI name to search for their
text description from Wikipedia website444https://www.wikipedia.org/. If there
is no direct match, we then retrieve the Wikipedia pages that mention the AOI
name and collect the related paragraphs. Each AOI record contains an AOI name,
a polygon geometry value, and a text description. For example, _[”Jamaica
Bay”, polygon (-73.86 40.58, …), ”Jamaica Bay is an estuary …”]_ is the record
of the AOI _”Jamaica Bay”_ with geometry value and text description.

Road Network Data. Road data describes the urban spatial transportation
network, including urban motorways, overpasses, and so on. We first follow
(Ning et al., 2023) to obtain the road name and geometry value from Open
Street Map (OSM)555https://www.openstreetmap.org/. Then, following the same
text acquisition operation in AOI data, we crawl the textual description of
each road record from Wikipedia. Each road record contains a road name, a
linestring geometry value, a road type and a text description. For example,
_[”Central Park Avenue”, linestring (-73.87 40.90, …), primary, ”Central Park
Avenue is a boulevard in …”]_ describes the primary road named _”Central Park
Avenue”_ with a linestring geometry value and its textual description.

Point-Of-Interest (POI) Data. POI data represents different urban functions
(e.g., residential and commercial), which have been widely adopted in many
recent UrbanKG works (Liu et al., 2023; Wang et al., 2021; Ning et al., 2023).
We first follow (Ning et al., 2023) to obtain the POI name, and geometry value
from OSM. Then the textual description of each POI record could be crawled
from Wikipedia following the similar process. Each POI record contains a POI
name, a coordinate geometry, a POI type, and a text description. For example,
_[”Trump World Tower”, coordinate (-73.96 40.75), residential, ”Trump World
Tower is a residential condominium …”]_ is the record of the POI _”Trump World
Tower”_.

####  2.1.2. Text Data

The text data provides rich contextual knowledge of the city space from
different perspectives (e.g., the spatial context) (Deng et al., 2023), and it
plays an important role in geospatial understanding. In this work, we collect
two types of text corpus.

Review Data. The review of urban places provides useful commercial information
that citizens use to make business decisions (Cho et al., 2011), which plays a
critical role in urban knowledge distillation. We collect review data from
Google Map666https://www.google.com/maps. Specifically, we first manually
split the city into multiple rectangular regions, then we utilize the Google
Map API to query the places contained within each region and their reviews.
Each review record contains a place name, a coordinate geometry value, a
rating, and a text review. For example, _[”Lifestyles Academy Inc”, coordinate
(-87.87 41.65), 4.9, ”Very nice organization and …”]_ is the review record of
place _”Lifestyles Academy Inc”_.

Web Page Data. The web page data works as the general text corpus for the
city, and it contains rich geoscience knowledge that has been utilized in
recent urban entity and relation extraction studies (Deng et al., 2023). We
collect web page data from the Google search engine. Specifically, we first
input the name of the crawled AOI, Road, and POI record into Google. Then we
concatenate the textual sentences of the top 10 retrieved web pages. Each web
page record contains a long urban text description.

Table 1. The statistics of raw datasets.  Dataset | Description | New York City | Chicago
---|---|---|---
Geographic Data | # of AOI | 183 | 80
# of road | 6,397 | 1,893
# of POI | 5,369 | 5,658
Text Data | # of review | 16,360 | 13,627
# of web page | 11,596 | 7,283

###  2.2. Data Preprocessing

Before constructing the UrbanKGC dataset, we first preprocess the raw
datasets. Specifically, we filter out AOIs, roads, POIs, reviews, and web
pages whose crawled textual descriptions are null value, too short (e.g., less
than ten word description) or meaningless (e.g., just repeating the POI name).
In addition, we remove irrelevant information from the text description, such
as non-English characters, non-ASCII gibberish, website addresses, and so on.
More details can be found in Appendix A.

##  3\. Preliminary

This section presents the UrbanKGC task definition and provides task analysis.

###  3.1. Task Definition and Problem Formulation

Before diving into the technical details, we first introduce the definition of
urban knowledge graph (UrbanKG):

###### Definition 1.

UrbanKG. The UrbanKG is defined as a multi-relational graph $\mathcal{G}$ =
($\mathcal{E}$, $\mathcal{R}$, $\mathcal{F}$), where $\mathcal{E}$,
$\mathcal{R}$ and $\mathcal{F}$ is the set of urban entities, relations and
facts, respectively. In particular, facts are defined as
$\mathcal{F}=\\{\langle h,r,t\rangle\mid
h,t\in\mathcal{E},r\in\mathcal{R}\\}$, where each triplet ${\langle
h,r,t\rangle}$ describes head entity ${h}$ is connected with tail entity ${t}$
via relation ${r}$.

The UrbanKG encodes diverse urban semantic knowledge by connecting urban
entities into a multi-relational graph. This work aims to construct an UrbanKG
from collected unstructured text data. We decompose the UrbanKG construction
(UrbanKGC) process into two sequential knowledge graph construction tasks,
namely relational triple extraction (Zheng et al., 2021) and knowledge graph
completion (Yao et al., 2023). We first provide the basic definition for these
two subtasks, and then introduce the problem formulation of this work.

####  3.1.1. Task Definition

We give the basic task definition as follow.

Relational Triplet Extraction (RTE). Given the unstructured texts, this task
achieves joint extraction of entities and their relations (Zheng et al., 2021)
which are in the form of a triplet ${\langle h,r,t\rangle}$. For instance,
given the urban text sentence _”Columbia University is a private Ivy league
research university in New York City.”_ , this task aims to identify two
entities _”Columbia University”_ and _”New York City”_ and their relation
_”locate-in”_ , described as triplet _¡Columbia University, locate-in, New
York City¿_.

Knowledge Graph Completion (KGC). Given a head entity $h$ and a tail entity
$t$, this task is to predict the missing relation between them (Yao et al.,
2023). For instance, given the head entity _”Columbia University”_ and the
tail entity _”Empire State Building”_ , this task is to predict that their
missing relation, e.g., _”disconnected”_ , described as triplet _¡Columbia
University, disconnected, Empire State Building¿_.

####  3.1.2. Problem Formulation

Given the urban unstructured text data, the desired output is an UrbanKG
$\mathcal{G}$. In this paper, this problem is decomposed into two sequential
subtasks: (1) Relational Triplet Extraction: the first task extracts
relational triplet ${\langle h,r,t\rangle}$ from the urban text data. The
output of RTE task is $\mathcal{G}_{1}$ =($\mathcal{E}$, $\mathcal{R}_{1}$,
$\mathcal{F}_{1}$), where $\mathcal{E}$ and $\mathcal{R}_{1}$is the set of
extracted entities and relations, while $\mathcal{F}_{1}$ is the set of
extracted triplets. (2) Knowledge Graph Completion: for the given head entity
$h$ and tail entity $t$ in $\mathcal{G}_{1}$, the second task is to predict
the geospatial relationship777We follow GeoLM (Li et al., 2023c) to provide
five RCC relationship (Mani et al., 2010) candidates: Disconnection (DC),
external connection (EC), equality (EQ), partial overlap (PO), and tangential
and non-tangential proper parts (IN). Details are in Appendix A. between them.
The output of this task is $\mathcal{G}_{2}$ =($\mathcal{E}$,
$\mathcal{R}_{2}$, $\mathcal{F}_{2}$), where $\mathcal{R}_{2}$ and
$\mathcal{F}_{2}$ is the set of completed relations and triplets. By
sequentially completing the above two tasks, we can obtain the constructed
UrbanKG $\mathcal{G}$ = ($\mathcal{E}$, $\mathcal{R}_{1}\cup\mathcal{R}_{2}$,
$\mathcal{F}_{1}\cup\mathcal{F}_{2}$).

###  3.2. Quantitative Task Analysis

Figure 3. Quantitative performance analysis of prompting GPT-4 for RTE and KGC
tasks. The result is obtained by comparing 50 GPT-4’s outputs with the human’s
annotation.

As shown in Figure 2, we qualitatively find that LLMs lack urban heterogeneous
relationship understanding ability and experience in geospatial computing and
reasoning difficulty when adopting it for UrbanKGC tasks. This subsection
presents a quantitative analysis of these two challenges.

Heterogenous Relationship Understanding. The ability to understand
heterogeneous relationships is ubiquitous in distilling knowledge from the
massive urban corpus. For example, the text description in Figure 2
illustrates a place from spatial location, temporal time, and functional
aspects. Capturing these heterogeneous semantics is important for urban
knowledge distillation. We perform quantitative analysis by randomly sampling
50 urban text data and then prompt GPT-4 to complete relational triplet
extraction by providing only the basic task description. As shown in Figure
3(a), we find that the LLMs experience serious misjudgment (i.e., extract
wrong triplets or miss the triplet) on urban spatial, temporal, and functional
triplet extraction. This indicates the limited capacity of LLMs to understand
heterogeneous relationships.

Geospatial Computing and Reasoning. Geospatial computing and reasoning
techniques are widely used in many previous UrbanKG studies (Ning et al.,
2023; Liu et al., 2023) for urban geospatial relation extraction. In recent
works (Mooney et al., 2023; Manvi et al., 2023), the geospatial skills of LLMs
have also been demonstrated to lack geospatial awareness and reasoning ability
(Bhandari et al., 2023). To identify potential limitations, we quantitatively
investigate how LLMs can perform on geospatial relation completion tasks.
Specifically, we construct 100 head and tail entity pairs, covering five
geospatial relations in the KGC task, and then prompt GPT-4 to predict with
basic task description and geospatial relation candidates. As shown in Figure
3(b), we find that GPT-4 performs poorly on five geospatial relation
completion. This further validates the disability of LLMs in geospatial
computing and reasoning.

##  4\. UrbanKGC Agent Construction

This section presents the proposed UrbanKGC agent construction framework.

###  4.1. Overview

The overall pipeline of the UrbanKGent framework is illustrated in Figure 4,
which consists of three major components: _(1) Knowledgeable Instruction
Generation_ consists of the heterogeneity-aware and geospatial-infused
instruction generation modules for aligning LLMs to UrbanKGC tasks. _(2) Tool-
augmented Iterative Trajectory Refinement_ proposes geospatial tool interface
invocation and iterative self-refinement mechanisms to enhance and refine
generated trajectory. _(3) Hybrid Instruction Fine-tuning_ fine-tune LLMs
based on the refined trajectories for cost-effectively completing diverse
UrbanKGC tasks.

###  4.2. Knowledgeable Instruction Generation

We first construct the knowledgeable instruction to adopt LLMs for two
UrbanKGC tasks, including relational triplet extraction (RTE) and knowledge
graph completion (KGC). Figure 4(a) illustrates the overview of the
instruction construction process of these two tasks.

####  4.2.1. Heterogeneity-aware Instruction Generation for Relational Triplet
Extraction.

As discussed in Section 3, the urban text contains diverse heterogeneous
relationships, thus we consider multiple views with both urban entity and
relation definition for relational triplet extraction. In particular, we
construct a multi-view instruction template for the urban relational triplet
extraction, including spatial view, temporal view, and functional view. Each
view is a multi-turn question-answer dialog (Wei et al., 2023) consisting of
entity recognition, relation extraction, and triplet extraction module.

For the spatial view, we devise a two-turn dialog to align LLMs for spatial
triplet extraction. In the first turn, we inject spatial entity and relation
definition into the instruction template to guide LLMs to understand spatial
characteristics and then extract potential spatial entities (e.g.,
_University_) and relations types (e.g., _locate-in_). In the second turn, the
extracted types are explicitly fed into the instruction template for spatial
triplet extraction. Intuitively, the spatial view allocates dedicated urban
knowledge for LLMs to extract urban spatial relationships. Similarly, we
construct the temporal view and functional view for corresponding temporal and
functional triplet extraction, independently. We provide the detailed
instruction template in the Appendix B.

####  4.2.2. Geospatial-infused Instruction Generation for Knowledge Graph
Completion.

Despite heterogeneity-aware instruction enabling LLMs to extract urban
triplets from various perspectives, the geospatial relationship between
geospatial entities cannot be directly extracted. Therefore, we introduce a
geospatial-infused instruction generation module to guide LLMs to complete
missing geospatial relationships.

First, we incorporate the geometry information (i.e., the latitude and
longitude) of geo-entities into instruction, so that the LLMs can utilize
these geospatial values for relation inference. Second, we add the geospatial
relationship definition to the instruction to guide LLMs in understanding the
geospatial relationship definition. Intuitively, LLMs can refer to this
geospatial knowledge and make practical solutions for the knowledge graph
completion task. We provide the detailed instruction template in the Appendix
B.

###  4.3. Tool-augmented Iterative Trajectory Refinement

Then, we present the trajectory generation to further augment and refine the
constructed UrbanKGC instructions.

####  4.3.1. Trajectory Generation

With the initial UrbanKGC instructions constructed, the following step is to
generate reasoning trajectories (Zeng et al., 2023), which will be used to
fine-tune LLMs tailored to UrbanKGC task. Specifically, we follow FireAct
(Chen et al., 2023) and use Chain-of-Thought (CoT) (Wei et al., 2022), a
gradient-free technique, to prompt GPT-4 (i.e., add prompt trigger _”Let’s
think step by step”_ at the end of RTE and KGC instructions template) to
generate the reasoning trajectories for UrbanKGC tasks.

The generated CoT trajectories could provide a step-by-step reasoning solution
for UrbanKGC tasks. Nevertheless, the complex geospatial relationships cannot
be easily extracted as discussed in Section 3 and recent geospatial reasoning
works (Manvi et al., 2023; Mooney et al., 2023; Bhandari et al., 2023).
Therefore, we introduce a tool invocation module to guide LLMs to invoke
tailored external geospatial tools (Cai et al., 2023) to enhance their
geospatial computing and reasoning capacity for UrbanKGC tasks.

Figure 4. An overview of UrbanKGent Construction.

####  4.3.2. Tool Invocation for Trajectory Augmentation

We conduct two sequential procedures: tool invocation for geospatial computing
support and trajectory deliberation for reasoning enhancement.

Tool Invocation. First, we construct a geospatial reasoning toolkit (e.g.,
distance calculation, eight interfaces in total shown in Table 8) by prompting
GPT-4 for self-programming. Then, we construct tailored prompts to guide LLMs
to invoke these interfaces. Specifically, the prompt is concatenated with an
illustrative description of the function of each geospatial tool and a task
instruction (i.e., _”Which types of tool interface you need”_). Intuitively,
the external tool allocates necessary calculation results for LLMs to infer
missing geospatial relation. The detailed toolkit description can be found in
Appendix C.

Trajectory Deliberation. After manipulation with external tools, we prompt
LLMs to refine uncertain reasoning steps based on these obtained manipulation
results. Specifically, we construct the prompt by concatenating with
manipulation results (e.g., the distance and geohash value of geo-entity) and
a task instruction (i.e., _”Please refine your reasoning process”_). After
feeding the prompt into GPT-4, the enhanced trajectory is obtained. Detailed
prompt information can be found in Appendix C.

####  4.3.3. Iterative Trajectory Self-refinement

Despite tool-augmented deliberation improving the geospatial computing and
reasoning ability of LLMs, enhanced trajectories may not all be faithful (Hong
et al., 2023). To alleviate potential error and ensure the trajectory quality
(Huang et al., 2023), we refine these trajectories via an iterative self-
refinement mechanism (Madaan et al., 2023). Specifically, we iterate two
sequential blocks: (i) Trajectory verifier: given the trajectory, the verifier
aims to provide feedback for refining the reasoning process; (ii) Trajectory
updater: given the trajectory and feedback, the updater will further refine
the current trajectory based on the feedback.

Trajectory Verifier. We construct a tailored prompt to ask LLMs to generate
feedback. Specifically, we use a simple but effective trigger (_”Judge whether
all extracted triplets are correct and provide improvement suggestion”_) to
prompt LLMs to provide feedback. If the trajectory no longer requires
modification, we let LLMs respond with _”This is a faithful trajectory”_. Such
a verification step lets LLMs make reflections and improve the correctness of
the trajectory.

Trajectory Updater. The updater utilizes provided feedback to refine the
current trajectory via prompt trigger _”Follow suggestion to refine the
reasoning process”_. Intuitively, the feedback may address multiple aspects
(e.g., missed triplet in the RTE task or unfaithful reasoning process in the
KGC task) of the unfaithful trajectories.

We iterate the trajectory verifier and updater until the predefined stopping
condition is satisfied. The stopping condition is determined by either meeting
the maximum number of iterations (we set it at three to avoid excessive cost)
or when the verifier confirms all trajectories are faithful. Upon meeting the
stopping condition, we use the last refined trajectory for further fine-
tuning. Detailed prompt information can be found in Appendix C.

###  4.4. Hybrid Instruction Fine-Tuning

To construct a cost-effective UrbanKGC agent, we further utilize trajectories
(generated by GPT-4) to fine-tune a smaller open-source LLM for faster
inference speed and lower cost (i.e., prompting GPT-4 for UrbanKGC is
expensive). Specifically, we finetune the LLM via the mixed-task instruction-
tuning strategy (Zeng et al., 2023). The goal is to enhance the LLMs’
capabilities in diverse UrbanKGC tasks.

####  4.4.1. Mixture Training

Set the base language model as $\mathcal{M}$, and
$\mathcal{P}_{\mathcal{M}}\left(y\mid x\right)$ represents the probability
distribution of response $y$ given instruction $x$. We consider the trajectory
set on two UrbanKGC tasks, i.e., $\mathcal{D}_{RTE}$ and $\mathcal{D}_{KGC}$.
Since both the instruction and the target output are formatted in natural
language, we can unify the training into an end-to-end sequence-to-sequence
way. Formally, the optimization process aims to minimize the loss of language
model $\mathcal{M}$ as follows:

(1)
$\displaystyle\mathcal{L}=\mathbb{E}_{(x,y)\sim\mathcal{D}_{\text{RTE}}}\left[\log\mathcal{P}_{\mathcal{M}}(y\mid
x)\right]+\mathbb{E}_{(x,y)\sim\mathcal{D}_{\text{KGC
}}}\left[\log\mathcal{P}_{\mathcal{M}}(y\mid x)\right],$

where $x$ and $y$ represent the instruction input and instruction output in
the trajectory, respectively.

####  4.4.2. Training Setup

We choose the chat version of open-sourced Llama 2-7/13B (Touvron et al.,
2023) as our backbone models, and fine-tune Llama using Low-Rank Adaptation
(LoRA) strategy (Hu et al., 2021). Through LoRA, the training process could be
done by tuning only one-thousandth of full LLM parameters.

###  4.5. Inference on UrbanKGC Task

Via hybrid instruction fine-tuning, the LLM can be trained to follow the
instructions to finish the UrbanKGC task. After obtaining the UrbanKGent, we
can prompt it to complete UrbanKGC tasks by following the pipeline shown in
Figure 4.

For the relational triplet extraction task, we sequentially execute entity
recognition, relation extraction, and relational triplet instruction
generation. We then perform iterative self-refinement and finally output the
extracted triplets. For the knowledge graph completion task, we sequentially
execute KGC instruction generation, external tool augmentation, iterative
self-refinement block, and finally output the completed triplets.

Table 2. The statistics of constructed UrbanKGC dataset. Dataset | NYC-Instruct | NYC | CHI-Instruct | CHI
---|---|---|---|---
Records | RTE | 232 | 2,089 | 122 | 1,102
KGC | 232 | 2,080 | 122 | 1,101

##  5\. Experiments

We evaluate the proposed method on two real-world UrbanKGC datasets, and aim
to answer the following research questions:

  * •

RQ1: How does the constructed UrbanKGent perform compared with existing
baselines and LLM prompting paradigms on UrbanKGC tasks?

  * •

RQ2: How do different components (e.g., the knowledgeable instruction
generation) affect the performance?

  * •

RQ3: How can the constructed UrbanKGent provide application service to real-
world scenarios?

Table 3. The main results of relational triplet extraction (RTE) and knowledge
graph completion (KGC). We report the accuracy (acc) and confidence for GPT
evaluation on two datasets, and report accuracy (acc) for the Human evaluation
approach. The best baseline performance is underlined.

Type | Models | NYC | CHI
---|---|---|---
GPT (acc/confidence) | Human (acc) | GPT (acc/confidence) | Human (acc)
RTE | KGC | RTE | KGC | RTE | KGC | RTE | KGC
Pretrained Language Models | KG-BERT | - | 0.24/3.15 | - | 0.23 | - | 0.19/4.12 | - | 0.24
KG-T5 | - | 0.21/4.02 | - | 0.21 | - | 0.15/3.98 | - | 0.24
RelationPrompt | 0.12/3.38 | - | 0.12 | - | 0.21/3.53 | - | 0.18 | -
Zero-shot Reasoning | Llama-2-7B | 0.14/1.98 | 0.18/3.75 | 0.16 | 0.18 | 0.26/1.96 | 0.15/2.83 | 0.21 | 0.22
Llama-2-13B | 0.21/2.07 | 0.28/3.91 | 0.19 | 0.22 | 0.22/2.19 | 0.16/2.47 | 0.22 | 0.24
Llama-2-70B | 0.25/3.07 | 0.28/3.75 | 0.22 | 0.24 | 0.27/3.55 | 0.16/2.47 | 0.24 | 0.23
GPT-3.5 | 0.29/4.11 | 0.36/3.47 | 0.31 | 0.23 | 0.31/3.79 | 0.31/3.16 | 0.31 | 0.29
GPT-4 | 0.38/4.03 | 0.39/3.82 | 0.41 | 0.29 | 0.39/4.08 | 0.32/4.03 | 0.43 | 0.35
In-context Learning | Llama-2-7B (1-shot) | 0.12/2.07 | 0.19/3.91 | 0.15 | 0.16 | 0.21/2.07 | 0.16/3.09 | 0.20 | 0.19
Llama-2-7B (3-shot) | 0.18/2.15 | 0.21/3.96 | 0.19 | 0.18 | 0.25/2.44 | 0.18/3.27 | 0.23 | 0.20
Llama-2-13B (1-shot) | 0.22/2.45 | 0.26/3.87 | 0.20 | 0.19 | 0.24/3.09 | 0.15/2.98 | 0.21 | 0.23
Llama-2-13B (3-shot) | 0.26/3.52 | 0.31/3.28 | 0.23 | 0.24 | 0.28/2.65 | 0.21/2.53 | 0.25 | 0.26
GPT-3.5 (1-shot) | 0.32/4.25 | 0.38/3.87 | 0.37 | 0.28 | 0.33/3.88 | 0.30/4.17 | 0.36 | 0.34
GPT-3.5 (3-shot) | 0.41/4.65 | 0.42/4.08 | 0.42 | 0.31 | 0.36/4.24 | 0.36/4.23 | 0.39 | 0.36
Vanilla Fine-tuning | Llama-2-7B | 0.32/4.37 | 0.38/3.65 | 0.32 | 0.27 | 0.29/3.80 | 0.30/3.65 | 0.33 | 0.31
Llama-2-13B | 0.35/4.26 | 0.41/3.92 | 0.39 | 0.29 | 0.31/4.14 | 0.29/3.87 | 0.37 | 0.35
UrbanKGent Inference | Llama-2-7B | 0.27/3.05 | 0.26/4.12 | 0.28 | 0.24 | 0.27/2.87 | 0.24/3.54 | 0.26 | 0.29
Llama-2-13B | 0.31/3.87 | 0.32/3.56 | 0.35 | 0.27 | 0.28/3.24 | 0.26/3.28 | 0.31 | 0.32
Llama-2-70B | 0.33/4.28 | 0.35/4.27 | 0.33 | 0.29 | 0.29/3.80 | 0.28/4.01 | 0.32 | 0.34
GPT-3.5 | 0.43/4.12 | 0.46/3.88 | 0.43 | 0.34 | 0.40/4.21 | 0.39/3.87 | 0.46 | 0.41
GPT-4 | 0.45/4.08 | 0.48/4.02 | 0.47 | 0.42 | 0.46/4.17 | 0.41/4.35 | 0.52 | 0.43
UrbanKGent-7B | 0.46/4.12 | 0.49/3.97 | 0.48 | 0.44 | 0.49/4.28 | 0.43/4.58 | 0.54 | 0.45
$\uparrow 2.22\%$ | $\uparrow 2.08\%$ | $\uparrow 2.08\%$ | $\uparrow 4.76\%$ | $\uparrow 6.52\%$ | $\uparrow 4.88\%$ | $\uparrow 3.84\%$ | $\uparrow 4.66\%$
UrbanKGent-13B | 0.52/4.38 | 0.56/4.13 | 0.54 | 0.47 | 0.53/4.15 | 0.48/4.42 | 0.59 | 0.49
$\uparrow 15.56\%$ | $\uparrow 14.29\%$ | $\uparrow 14.89\%$ | $\uparrow 11.90\%$ | $\uparrow 15.22\%$ | $\uparrow 17.07\%$ | $\uparrow 13.46\%$ | $\uparrow 13.95\%$

###  5.1. Experimental Settings

####  5.1.1. Dataset

In this work, two sequential tasks (i.e., RTE and KGC) of UrbanKGC are within
an open-world setting (i.e., no predefined ontology) (Lu et al., 2023; Ye et
al., 2022). Therefore, for the RTE task, every data record is an urban text
without the triplet label. For the KGC task, every data record is a quadruple
(i.e., head entity name, head entity geometry, tail entity name, tail entity
geometry) without the geospatial relation label. We construct the RTE and KGC
datasets of NYC and CHI by sampling uniformly from five raw data in Table 1,
respectively. As shown in Table 2, we first construct two small datasets
(i.e., NYC-Instruct and CHI-Instruct) for instruction fine-tuning and two
middle datasets (i.e., NYC and CHI) to validate the performance of the
constructed UrbanKGC agent. The remaining data works as the large-scale
UrbanKGC dataset (i.e., NYC-Large and CHI-Large) in real-world scenarios shown
in Table 7. The three types of datasets are non-overlapping to prevent data
leakage. More dataset construction details are in Appendix A.

For each UrbanKGC dataset, we follow Section 4.5 to prompt constructed
UrbanKGent to complete RTE and KGC tasks.

####  5.1.2. Baseline Methods

We provide a comprehensive comparison of our method with existing paradigms
(e.g., zero-shot reasoning (Wang et al., 2023a), in-context learning (Wei et
al., 2022) and vanilla finetuning (Yao et al., 2023)) on the UrbanKGC tasks.

Pretrained Language Model Baselines. For the RTE task, we utilize
RelationPrompt (Chia et al., 2022), an end-to-end generation model for zero-
shot RTE. For the KGC task, we fine-tune KG-BERT (Yao et al., 2019) and KG-T5
(Saxena et al., 2022) with the QA pairs constructed from the self-instruct
dataset.

LLMs-based Zero-shot Reasoning Methods. We directly prompt the LLMs with basic
task definitions to get the answer without training.

LLMs-based In-context Learning Methods. We sample a few-shot QA pairs as
demonstrations from the self-instruct dataset as examples and get the answers
from the LLMs without training.

Vanilla Fine-tuning Methods. We directly fine-tune the LLMs using the QA pairs
constructed from the self-instruct dataset, and then prompt the LLMs with
basic task definition without demonstrations.

UrbanKGent Inference Methods. We directly prompt the LLMs using the
UrbanKGgent inference pipeline in Section 4.5.

The prompt templates of the above baseline methods are in Appendix B.

####  5.1.3. Implementation and Detail Settings

In our experiment, we select Llama-2, GPT-3.5 and GPT-4 as the backbone LLM
$\mathcal{M}$. We follow the open-source Llama
document888https://ai.meta.com/llama/ to deploy Llama-2-7B, Llama-2-13B, and
Llama-2-70B in our device. We implement two fine-tuning agent versions (e.g.,
Llama-2-7B and Llama-2-13B), and all experiments are conducted on eight NVIDIA
A800 GPUs. For the GPT-3.5 and GPT-4, we adopt the gpt-3.5-turbo-16k-0613 API
and gpt-4-0613 API from the GPT service999https://gpt.hkust-gz.edu.cn/
provided by HKUST (GZ).

####  5.1.4. Evaluation Protocol

Since the UrbanKGC tasks in this work follow an open-world setting where
labels are not visible, the classical metric (e.g., F1 and Hits@10) is not
applicable. In this work, we regard evaluation as the binary classification,
i.e., if the extracted triplet in RTE task is correct and if the completed
relation in KGC task is correct. We follow recent LLMs-based KGC works (Yao et
al., 2023) to employ accuracy as an evaluation metric. To make a comprehensive
evaluation of the experimental results, we employ the evaluation from two
aspects:

Human Evaluation. We employ human annotators to evaluate the results on 200
random samples. For the relational triplet extraction task, we first manually
annotate the triplet label for each sample. Then, we manually evaluate the
correctness of each triplet (Lu et al., 2023) based on annotation and
calculate the accuracy value. For the knowledge graph completion task, we
follow (Yao et al., 2023) to manually label the response as correct or wrong,
and calculate the accuracy. The detailed human evaluation process is in
Appendix D.

GPT Evaluation. Recently, many studies (Zheng et al., 2023; Wang et al.,
2023a) adopt LLM-based evaluation for open-domain tasks and empirically
demonstrate that GPT-4’s evaluation and human evaluation can be consistent
(Cheng et al., 2023). In this work, we also use GPT-4 to evaluate the model
performance on the full data to escape intensive labor. Specifically, given an
UrbanKGC instruction and results, we prompt GPT-4 to return the confidence
score and the justification (i.e., True/False), which will be further used to
calculate the accuracy. In this work, the GPT-4’s evaluation has also been
demonstrated to be consistent with the human evaluation. The detail is in
Appendix D.

Table 4. Effect of different blocks. Models | GPT (acc/confidence) | Human (acc)
---|---|---
RTE | KGC | RTE | KGC
UrbanKGent-7B♠ | 0.38/4.17 | 0.42/3.98 | 0.37 | 0.34
UrbanKGent-7B⋆ | 0.34/4.06 | 0.45/4.02 | 0.34 | 0.39
UrbanKGent-7B‡ | 0.45/4.32 | 0.40/3.97 | 0.45 | 0.23
UrbanKGent-7B† | 0.44/4.10 | 0.47/3.85 | 0.46 | 0.43

###  5.2. Main Result (RQ1)

The performance results are reported in Table 3. As can be seen, the
constructed agent outperforms all twenty-one baseline models on two UrbanKGC
datasets. Specifically, the UrbanKGent-13B achieves (15.56%, 14.29%, 14.89%,
and 11.90%) improvements compared with the state-of-the-art GPT-4 with the
same inference pipeline on NYC. The improvements on CHI are (15.22%, 17.07%,
13.46%, and 13.95%), respectively. Moreover, the UrbanKGent-7B also achieves
comparable performance compared with the GPT-4. Such improvement demonstrates
the superiority of the constructed UrbanKGC agent.

Meanwhile, we observe that the zero-shot LLMs perform poorly in the UrbanKGC
tasks, even using GPT-4. In addition, although the demonstrations provided by
In-context-learning can incorporate the UrbanKGC task information, the
performance gain is limited and even leads to performance degradation in
smaller LLMs (e.g., Llama-2-7/13B). Besides, we find that fine-tuning LLMs can
make obvious improvements in the overall performance. Through vanilla fine-
tuning, the Llama-2-7B and Llama-2-13B could achieve comparable performance
with GPT-3.5 under the zero-shot reasoning and In-context learning settings.

Moreover, although the LLMs (i.e., Llama-2-7B and Llama-2-13B) using the
UrbanKGent inference pipeline perform slightly worse than the vanilla fine-
tuning method, they could obtain better performance compared with zero-shot
reasoning and In-context learning paradigms. Such results demonstrate the
benefit of knowledgeable instruction design and external tool innovation, but
also indicate its performance bottleneck (i.e., well-designed UrbanKGent
prompting pipeline unable to beat vanilla fine-tuning). As a deeper
exploration, our work fills this gap through hybrid instruction fine-tuning,
and the fine-tuned UrbanKGC agents, whether 7B or 13B, can achieve state-of-
the-art performance in UrbanKGC tasks.

Table 5. Comparison among LLM-based UrbanKGC methods in four ways.

Method |  | Extra
---
Knowledge
| Require
---
Fine-tuning
| Tool
---
Invokation
| Self
---
Refinement
| ZSL
---
$\times$ | $\times$ | $\times$ | $\times$
| ICL
---
$\surd$ | $\times$ | $\times$ | $\times$
| VFT
---
$\surd$ | $\surd$ | $\times$ | $\times$
| UrbanKGent
---
Inference
$\surd$ | $\times$ | $\surd$ | $\surd$
UrbanKGent | $\surd$ | $\surd$ | $\surd$ | $\surd$

###  5.3. In-Depth Analysis (RQ2)

We conduct an in-depth analysis of the proposed instruction generation and
tool-augmented iterative trajectory refinement module on the NYC dataset.
Specifically, for the RTE and KGC task, we validate the effectiveness of each
block by comparing the following variants: (1) UrbanKGent-7B♠ removes
knowledgeable instruction template in RTE and KGC task; (2) UKGent⋆ removes
multi-view design in RTE task; (3) UrbanKGent-7B‡ removes tool invocation; (4)
UrbanKGent-7B† removes iterative trajectory self-refinement. We summarize the
results in Table 4, and obtain the following observations.

First, knowledgeable instruction generation contributes to the overall
performance of both RTE and KGC tasks. We observe a performance degradation by
removing the knowledgeable instruction template. Second, the multi-view
instruction design provides the most performance gain, which matches our
intuition that the UrbanKG text contains heterogeneous relationships that can
be effectively extracted by multi-view prompting design. Third, the tool
invocation is very important for the KGC task, as we can observe significant
performance degradation after removing the tool invocation. In addition, the
iterative trajectory self-refinement brings consistent performance gain for
both the RTE and KGC tasks.

Figure 5. The model latency and cost of constructed UrbanKGent-13B and GPT-4
in UrbanKGC tasks. We report the total inference time and cost of 1,000 RTE
and KGC tasks.

###  5.4. Complexity and Effiency Analysis

We make a comparison with the four paradigms to demonstrate the advantages of
the constructed agent, which is shown in Table 5. Compared with Zero-shot
reasoning (ZSL), In-context Learning (ICL), Vanilla Fine-tuning (VFT), and
UrbanKGent Inference, UrbanKGent can incorporate extra urban knowledge, invoke
external tools and iteratively self-refine to help better complete UrbanKGC
tasks.

Moreover, we also provide efficiency analysis in Figure 5. As can be seen,
UrbanKGent-13B achieve lower inference speed in latency and reduce the cost by
roughly 20 times in both RTE and KGC tasks. The detail is in Appendix H.

###  5.5. Agent Application and Deployment (RQ3)

We have deployed a prototype
system101010https://htmlpreview.github.io/?https://raw.githubusercontent.com/usail-
hkust/UrbanKGent/main/UrbanKGent%20Demo/index.html to facilitate real-world
UrbanKGC construction. The service is equipped with the UrbanKGent-13B version
and is currently optimized for New York City and Chicago.

As shown in Table 6, compared to the previous UrbanKG benchmark (Ning et al.,
2023), we only use roughly one-fifth of the data for constructing the UrbanKGs
with the same scale of triplets and entities, and even expanding the variety
of relationships to a thousand times the original types. Please refer to
Appendix F for more details.

Table 6. Statistics comparision between constructed two UrbanKGs in NYC and
CHI and existing UrbanKG dataset and benchmark UUKG (Ning et al., 2023). For
relation and data volume, we report the increase and decrease percentages in
terms of absolute percentage returns.

Dataset | # Entity | # Relation | # Triplet | Data Volume
---|---|---|---|---
NYC-Large | 265,938 | 24,052 ($\uparrow 1850\%$) | 975,102 | 39,600 ($\downarrow 597\%$)
CHI-Large | 191,473 | 21,089 ($\uparrow 1622\%$) | 610,258 | 28,378 ($\downarrow 495\%$)
NYC-UUKG | 236,287 | 13 | 930,240 | 236,277
CHI-UUKG | 140,602 | 13 | 564,400 | 140,577

##  6\. Related work

Our work is related to domain-oriented agent construction, LLMs for knowledge
graph construction and urban knowledge graph. We briefly discuss them in this
section

###  6.1. Domain-Oriented Agent Construction

The concept of language agent (Chen et al., 2023) has become very popular
recently, and a variety of LLM agents targeting different domains have been
proposed. For example, Voyager (Wang et al., 2023c) is constructed for
automated game exploration, WebGPT (Gur et al., 2023) is an HTML agent for
diverse document understanding tasks, LLMLight (Lai et al., 2023) constructs a
language agent for transportation domain, K2 (Deng et al., 2023), GeoGalactica
(Lin et al., 2023) and GeoLLM (Li et al., 2023c) propose to re-train language
agent for geospatial semantic understanding. In addition, many recent works
like Auto-GPT (aut, 2023) and CAMEL (Li et al., 2023a) aim at proposing an
autonomous agent framework for agent construction. Nevertheless, there is
still no UrbanKGC agent construction framework for the urban computing domain.

###  6.2. LLMs for Knowledge Graph Construction

Knowledge Graph Construction aims to extract structural information from
unstructured text (Ye et al., 2022). Recently, the advent of Large Language
Models (LLMs) invigorated the field of Natural Language Processing (NLP). Many
studies have begun to explore the potential of LLMs in the domain of knowledge
graph construction. For example, (Wei et al., 2023; Li et al., 2023b) finds
that transforming the NER and RE task into a multi-turn question-answering
dialog could improve the performance of LLMs on the KG construction task. (Xie
et al., 2023) explicitly derive syntactic knowledge to guide LLMs to think,
which could develop the performance of NER. Despite these LLM-driven knowledge
graph construction methods (Yuan et al., 2023; Lu et al., 2023) in general
domains being widely investigated, knowledge graph construction in specific
domains still remains an open challenge (Zeng and Wang, 2023). There is still
no promising knowledge graph construction method in the urban computing
domain, and this hinders its potential achievement.

###  6.3. Urban Knowledge Graph

Urban knowledge graph has been proven useful in various urban tasks, such as
traffic flow prediction (Yang and Qi, 2022; Tan et al., 2021; Liu et al.,
2021; Zhao et al., 2020), mobility prediction (Wang et al., 2021), site
selection (Liu et al., 2022), city profiling (Zhou et al., 2023), crime
prediction and so on (Ning et al., 2023). Their common approach involves
manually extracting urban entities and defining urban relations to construct
an urban knowledge graph. For example, (Wang et al., 2021) construct a
dedicated spatiotemporal knowledge graph regarding trajectory and timestamp as
entities to improve trajectory prediction and (Liu et al., 2022) construct
user check-in relations to help mobility prediction. Nevertheless, existing
UrbanKGs heavily rely on manual design, leading to high labor costs and
constrained by a limited variety of urban entities and relation categories.

##  7\. Conclusion

In this work, we proposed UrbanKGent, the first automatic UrbanKG construction
agent framework with Large Language Models (LLMs). We first constructed a
knowledgeable instruction set to adopt LLMs for different UrbanKGC tasks.
Then, we proposed a tool-augmented iterative trajectory refinement module to
facilitate the instruction tuning of various large language models. Extensive
experimental results demonstrate the advancement of UrbanKGent in improving
UrbanKGC tasks. We release the UrbanKGent-13B, an agent based on Llama-13B,
with lower latency and cost compared with GPT-4 for UrbanKG construction. We
hope the open-source UrbanKGent can foster future urban knowledge graph
research and broader smart city applications.

##  8\. Limitation and Future Work

This work has limitation on the further application demonstration of
construction UrbanKGs, although proposed UrbanKGent-13B could construct a
UrbanKG with a thousandfold relationship using only one-fifth of data. In
addition, the evaluation method in this work is cost-intensive although GPT
evaluation and Human evaluation has been experimentally demonstrated to be
consistent. Despite the above limitations, we hope the opensource UrbanKGC
agent can foster more extensive UrbanKG research and broad smart city
application. In the future, we will derive extra image-modality data to
further enrich UrbanKGC.

## References

  * (1)
  * aut (2023) 2023\.  Significant-gravitas/auto-gpt: An experimental open-source attempt to make gpt-4 fully autonomous.  (2023).
  * Bhandari et al. (2023) Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser. 2023.  Are Large Language Models Geospatially Knowledgeable?. In _Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems_. 1–4.
  * Cai et al. (2023) Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. 2023.  Large language models as tool makers.  _arXiv preprint arXiv:2305.17126_ (2023).
  * Chen et al. (2023) Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023.  Fireact: Toward language agent fine-tuning.  _arXiv preprint arXiv:2310.05915_ (2023).
  * Chen et al. (2016) Quanjun Chen, Xuan Song, Harutoshi Yamada, and Ryosuke Shibasaki. 2016.  Learning deep representation from big and heterogeneous data for traffic accident inference. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_. 338–344.
  * Cheng et al. (2023) Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, et al. 2023\.  Evaluating hallucinations in chinese large language models.  _arXiv preprint arXiv:2310.03368_ (2023).
  * Cheng et al. (2018) Weiyu Cheng, Yanyan Shen, Yanmin Zhu, and Linpeng Huang. 2018.  A neural attention model for urban air quality inference: Learning the weights of monitoring stations. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 2018, USA, February 2-7, 2018_. 2151–2158.
  * Chia et al. (2022) Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022.  RelationPrompt: Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction.  _arXiv preprint arXiv:2203.09101_ (2022).
  * Cho et al. (2011) Eunjoon Cho, Seth A Myers, and Jure Leskovec. 2011.  Friendship and mobility: user movement in location-based social networks. In _Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1082–1090.
  * Deng et al. (2023) Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. 2023\.  Learning A Foundation Language Model for Geoscience Knowledge Understanding and Utilization.  _arXiv preprint arXiv:2306.05064_ (2023).
  * et al (2023a) Wangchunshu Zhou et al. 2023a.  Agents: An Open-source Framework for Autonomous Language Agents.  _ArXiv_ abs/2309.07870 (2023).  https://api.semanticscholar.org/CorpusID:261822166
  * et al (2023b) Zhiheng Xi et al. 2023b.  The Rise and Potential of Large Language Model Based Agents: A Survey.  _ArXiv_ abs/2309.07864 (2023).  https://api.semanticscholar.org/CorpusID:261817592
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023.  Mathematical capabilities of chatgpt.  _arXiv preprint arXiv:2301.13867_ (2023).
  * Fu et al. (2023a) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023a.  Gptscore: Evaluate as you desire.  _arXiv preprint arXiv:2302.04166_ (2023).
  * Fu et al. (2023b) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023b.  Specializing Smaller Language Models towards Multi-Step Reasoning.  _arXiv preprint arXiv:2301.12726_ (2023).
  * Gur et al. (2023) Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. 2023.  A real-world webagent with planning, long context understanding, and program synthesis.  _arXiv preprint arXiv:2307.12856_ (2023).
  * Hong et al. (2023) Ruixin Hong, Hongming Zhang, Hong Zhao, Dong Yu, and Changshui Zhang. 2023.  Faithful Question Answering with Monte-Carlo Planning.  _arXiv preprint arXiv:2305.02556_ (2023).
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ (2021).
  * Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023.  Large language models cannot self-correct reasoning yet.  _arXiv preprint arXiv:2310.01798_ (2023).
  * Lai et al. (2023) Siqi Lai, Zhao Xu, Weijia Zhang, Hao Liu, and Hui Xiong. 2023.  Large Language Models as Traffic Signal Control Agents: Capacity and Opportunity.  _arXiv preprint arXiv:2312.16044_ (2023).
  * Li et al. (2023a) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a.  Camel: Communicative agents for” mind” exploration of large scale language model society.  _arXiv preprint arXiv:2303.17760_ (2023).
  * Li et al. (2023b) Guozheng Li, Peng Wang, and Wenjun Ke. 2023b.  Revisiting Large Language Models as Zero-shot Relation Extractors.  _arXiv preprint arXiv:2310.05028_ (2023).
  * Li et al. (2023c) Zekun Li, Wenxuan Zhou, Yao-Yi Chiang, and Muhao Chen. 2023c.  GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_. 5227–5240.
  * Lin et al. (2023) Zhouhan Lin, Cheng Deng, Le Zhou, Tianhang Zhang, Yi Xu, Yutong Xu, Zhongmou He, Yuanyuan Shi, Beiya Dai, Yunchong Song, et al. 2023\.  GeoGalactica: A Scientific Large Language Model in Geoscience.  _arXiv preprint arXiv:2401.00434_ (2023).
  * Liu et al. (2021) Jia Liu, Tianrui Li, Shenggong Ji, Peng Xie, Shengdong Du, Fei Teng, and Junbo Zhang. 2021.  Urban flow pattern mining based on multi-source heterogeneous data fusion and knowledge graph embedding.  _IEEE Transactions on Knowledge and Data Engineering_ (2021).
  * Liu et al. (2023) Yu Liu, Jingtao Ding, Yanjie Fu, and Yong Li. 2023.  UrbanKG: An Urban Knowledge Graph System.  _ACM Transactions on Intelligent Systems and Technology_ (2023), 1–25.
  * Liu et al. (2022) Yu Liu, Jingtao Ding, and Yong Li. 2022.  Developing knowledge graph based system for urban computing. In _Proceedings of the 1st ACM SIGSPATIAL International Workshop on Geospatial Knowledge Graphs_. 3–7.
  * Lu et al. (2023) Keming Lu, Xiaoman Pan, Kaiqiang Song, Hongming Zhang, Dong Yu, and Jianshu Chen. 2023.  PIVOINE: Instruction Tuning for Open-world Information Extraction.  _arXiv preprint arXiv:2305.14898_ (2023).
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023\.  Self-refine: Iterative refinement with self-feedback.  _arXiv preprint arXiv:2303.17651_ (2023).
  * Mani et al. (2010) Inderjeet Mani, Christy Doran, Dave Harris, Janet Hitzeman, Rob Quimby, Justin Richer, Ben Wellner, Scott Mardis, and Seamus Clancy. 2010.  SpatialML: annotation scheme, resources, and evaluation.  _Language Resources and Evaluation_ (2010), 263–280.
  * Manvi et al. (2023) Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, and Stefano Ermon. 2023.  Geollm: Extracting geospatial knowledge from large language models.  _arXiv preprint arXiv:2310.06213_ (2023).
  * Mooney et al. (2023) Peter Mooney, Wencong Cui, Boyuan Guan, and Levente Juhász. 2023.  Towards Understanding the Geospatial Skills of ChatGPT: Taking a Geographic Information Systems (GIS) Exam. In _Proceedings of the 6th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery_. 85–94.
  * Ning et al. (2023) Yansong Ning, Hao Liu, Hao Wang, Zhenyu Zeng, and Hui Xiong. 2023.  UUKG: Unified Urban Knowledge Graph Dataset for Urban Spatiotemporal Prediction.  _arXiv preprint arXiv:2306.11443_ (2023).
  * Saxena et al. (2022) Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022.  Sequence-to-sequence knowledge graph completion and question answering.  _arXiv preprint arXiv:2203.10321_ (2022).
  * Sun et al. (2017) Fangzhou Sun, Abhishek Dubey, and Jules White. 2017.  DxNAT—Deep neural networks for explaining non-recurring traffic congestion. In _2017 IEEE International Conference on Big Data (IEEE BigData 2017), Boston, MA, USA, December 11-14, 2017_. 2141–2150.
  * Tan et al. (2021) Jiyuan Tan, Qianqian Qiu, Weiwei Guo, and Tingshuai Li. 2021.  Research on the construction of a knowledge graph and knowledge reasoning model in the field of urban traffic.  _Sustainability_ (2021), 3191.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, and Kevin Stone et al. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _CoRR_ abs/2307.09288 (2023).  arXiv:2307.09288
  * Wang et al. (2023a) Cunxiang Wang, Sirui Cheng, Zhikun Xu, Bowen Ding, Yidong Wang, and Yue Zhang. 2023a.  Evaluating open question answering evaluation.  _arXiv preprint arXiv:2305.12421_ (2023).
  * Wang et al. (2023c) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023c.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ (2023).
  * Wang et al. (2021) Huandong Wang, Qiaohong Yu, Yu Liu, Depeng Jin, and Yong Li. 2021.  Spatio-temporal urban knowledge graph enabled mobility prediction.  _Proc. ACM Interact. Mob. Wearable Ubiquitous Technol._ , 184:1–184:24.
  * Wang et al. (2023b) Xudong Wang, Slam Nady, Zixiang Zhang, Mingtong Zhang, and Jingrong Wang. 2023b.  Knowledge Graph of Urban Firefighting with Rule-Based Entity Extraction. In _International Conference on Engineering Applications of Neural Networks_. Springer, 168–177.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 24824–24837.
  * Wei et al. (2023) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023\.  Zero-shot information extraction via chatting with chatgpt.  _arXiv preprint arXiv:2302.10205_ (2023).
  * Xie et al. (2023) Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023.  Empirical study of zero-shot ner with chatgpt.  _arXiv preprint arXiv:2310.10035_ (2023).
  * Yang and Qi (2022) Chengbiao Yang and Guilin Qi. 2022.  An urban traffic knowledge graph-driven spatial-temporal graph convolutional network for traffic flow prediction. In _Proceedings of the 11th International Joint Conference on Knowledge Graphs_. 110–114.
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  KG-BERT: BERT for knowledge graph completion.  _arXiv preprint arXiv:1909.03193_ (2019).
  * Yao et al. (2023) Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. 2023.  Exploring large language models for knowledge graph completion.  _arXiv preprint arXiv:2308.13916_ (2023).
  * Ye et al. (2022) Hongbin Ye, Ningyu Zhang, Hui Chen, and Huajun Chen. 2022.  Generative knowledge graph construction: A review.  _arXiv preprint arXiv:2210.12714_ (2022).
  * Yuan et al. (2023) Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023.  Zero-shot temporal relation extraction with chatgpt.  _arXiv preprint arXiv:2304.05454_ (2023).
  * Zar (2005) Jerrold H Zar. 2005.  Spearman rank correlation.  _Encyclopedia of biostatistics_ (2005).
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023.  Agenttuning: Enabling generalized agent abilities for llms.  _arXiv preprint arXiv:2310.12823_ (2023).
  * Zeng and Wang (2023) Yueling Zeng and Li-C Wang. 2023.  Domain Knowledge Graph Construction Via A Simple Checker.  _arXiv preprint arXiv:2310.04949_ (2023).
  * Zhang et al. (2023b) Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2023b.  Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning.  _arXiv preprint arXiv:2306.02408_ (2023).
  * Zhang et al. (2023a) Shiyuan Zhang, Tong Li, Shuodi Hui, Guangyu Li, Yanping Liang, Li Yu, Depeng Jin, and Yong Li. 2023a.  Deep transfer learning for city-scale cellular traffic generation through urban knowledge graph. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4842–4851.
  * Zhao et al. (2020) Ling Zhao, Hanhan Deng, Linyao Qiu, Sumin Li, Zhixiang Hou, Hai Sun, and Yun Chen. 2020.  Urban multi-source spatio-temporal data analysis aware knowledge graph embedding.  _Symmetry_ (2020), 199.
  * Zheng et al. (2021) Hengyi Zheng, Rui Wen, Xi Chen, Yifan Yang, Yunyan Zhang, Ziheng Zhang, Ningyu Zhang, Bin Qin, Ming Xu, and Yefeng Zheng. 2021.  PRGC: Potential relation and global correspondence based joint relational triple extraction.  _arXiv preprint arXiv:2106.09895_ (2021).
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023\.  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.  _arXiv preprint arXiv:2306.05685_ (2023).
  * Zhou et al. (2023) Zhilun Zhou, Yu Liu, Jingtao Ding, Depeng Jin, and Yong Li. 2023.  Hierarchical knowledge graph learning enabled socioeconomic indicator prediction in location-based social network. In _Proceedings of the ACM Web Conference_. 122–132.

##  Appendix A UrbanKGC Construction

Table 7. The detailed statistic of RTE datasets. We report the maximum length,
minimum length, and average length of urban text in the RTE dataset.

Dataset | Max Length | Min Length | Avg Length | # Records
---|---|---|---|---
NYC-Instruct | 1,747 | 68 | 437 | 232
CHI-Instruct | 1,120 | 25 | 408 | 122
NYC | 2,708 | 51 | 433 | 2,089
CHI | 1,883 | 32 | 445 | 1,102
NYC-Large | 4,277 | 28 | 1,167 | 39,600
CHI-Large | 4,306 | 36 | 804 | 28,378

This section presents detailed UrbanKGC dataset statistic information for
relational triplet extraction (RTE) and knowledge graph completion (KGC)
tasks.

To facilitate the understanding of the constructed RTE dataset, we summarize
the distribution of entity types in the four RTE datasets, and the text
statistics of entity textual descriptions. As shown in Table 7, the entity
distribution and textual statistics of the instruct dataset and test data are
similar. In addition, the record in dataset (i.e., NYC-Instruct and CHI-
Instruct) used for instruction tuning is not overlapping with that in test
dataset (i.e., NYC and CHI). This can avoid potential data leakage issues.

As for the KGC dataset, we provide illustrative visualization of the five RCC
relationships (Mani et al., 2010) in Figure 6 for better understanding.
Specifically, the disconnected (DC), externally connected (EC), partially
overlapping (PO), equal (EQ), tangential and non-tangential proper parts (IN)
together depict the basic geospatial relationship between urban entities.
Moreover, to facilitate the understanding of the constructed KGC dataset, we
visualize the geometry range of the head entity and the tail entity in four
KGC datasets. As can be seen in Figure 7, the entity distribution in the
instruct dataset (i.e., NYC-Instruct and CHI-Instruct) and test dataset (i.e.,
NYC and CHI) are similar. Due to the KGC task follows a zero-shot setting, we
are unable to provide accurate distribution proportions of the 5 RCC
relationships in the four datasets. However, it is intuitive can be seen that
all four datasets contain the five RCC relationships shown in Figure 6, which
provides a guarantee for the practical significance of KGC task in this work.

Figure 6. Given two geo-entities X and Y, the illustrative visualization of
five types of RCC relationships. In this work, we consider entities with
coordinate geometry to a circles with very small radii.

##  Appendix B Instruction Template

###  B.1. Instruction Template in UrbanKGent

This section presents the detailed instruction template of the proposed
UrbanKGent framework. Specifically, Figure 8(a-b) provides the detailed
instruction template of knowledgeable instruction generation module and tool-
based trajectory augmentation. The iterative trajectory self-refinement module
is achieved by trajectory verifier and updater in Figure 8(c).

Figure 7. The geometry range visualization of the head entity and tail entity
of four KGC datasets. The horizontal and vertical coordinates are longitude
and latitude, respectively. The blue and red polygons stand for entities with
the polygon geometry, the purple line string stands for the entities with
linestring geometry and the green point is for the coordinate entities.

###  B.2. Instruction Template in Baselines

We provide the detailed instruction template of all the baseline models in
this work.

LLMs-based Zero-shot Reasoning Methods. We only provide task descriptions for
the zero-shot reasoning method. The detailed instruction template is shown in
Figure 9.

LLMs-based In-context Learning Methods. We first construct several few-shot
demonstrations via the chain-of-thought prompting techniques (Wei et al.,
2022), which is popular for automatic demonstration generation. The detailed
instruction template is shown in Figure 9. Then, we add these demonstrations
before the test question, and the detailed instruction template is shown in
Figure 9.

Vanilla Fine-tuning Methods. We only provide task descriptions for vanilla
fine-tuning methods without demonstrations. The detailed instruction template
is shown in Figure 9.

UrbanKGent Inference Methods. These baseline models follow the same inference
pipeline shown in Figure 8.

Figure 8. The reasoning process of UrbanKGent-13B and the detailed instruction
template for UrbanKent inference pipeline. The content in the gray box is the
instruction, and the content in the green box is the agent’s response. Figure
9. The instruction template of zero-shot reasoning, In-context learning and
vanilla fine-tuning baselines. Table 8. The detailed geospatial tool name,
description, interface input, and output. Each tool interface is implemented
by python and it is self-programmed by GPT-4.

Tool name | Tool description | Input | Output
---|---|---|---
Geohash | Geohash encoding | Geometry | Geohash code (8-bit)
Distance | Calculate the distance between two geo entities. | Geometry 1, Geometry 2 | Distance value (km)
Point2Polygon | Identify if a point belongs to a polygon | Point geometry, polygon geometry | True/False
Point4Linestring | Identify if a point intersects a linestring | Point geometry, linestring geometry | True/False
Linestring2Polygon | Identify if a linestring belongs to a polygon | Linestring geometry, polygon geometry | True/False
Linestring4Polygon | Identify if a linestring intersects a polygon | Linestring geometry, polygon geometry | True/False
Polygon2Polygon | Identify if a polygon belongs to a polygon | Polygon geometry, polygon geometry | True/False
Polygon4Polygon | Identify if a polygon intersects a polygon | Polygon geometry, polygon geometry | True/False

##  Appendix C Geospatial Toolkit

Table 9. Illustrative RTE evaluation example when we utilize human evaluation
and GPT evaluation method. We calculate the accuracy by counting the
proportion of true triplets. The label for GPT evaluation method is invisible.

Type | Urban text | Results | Label |  | Number of the
---
true triplet
| Number of the
---
false triplet
| Human
---
Evaluation
Columbia University is a private Ivy league research university in New York City. |  <Columbia University, locate-in, New York City > |  | <Columbia University,
---
locate-in, New York City >
1 | 0
| GPT
---
Evaluation
- | 1 | 0

This work constructs a geospatial reasoning toolkit (eight interfaces in total
in Table 8) by asking GPT-4 for self-programming. We obtained geospatial
toolkit for point, linestring, and polygon geometry objects, supporting
distance calculation and Geohash encoding for two geographical objects, and
can also determine basic spatial relationships between two geometric objects,
such as containment and intersection. For each interface, we provide its name
and function description (e.g., _geohash encoding_ in Table 8 and Figure 8b)
into the instruction template.

##  Appendix D Evaluation

This section presents the detailed evaluation process and examples to help the
reader have a better understanding. Moreover, we provide a comprehensive
analysis to demonstrate why GPT and Human evaluations are highly aligned.

###  D.1. Human Evaluation Process

For the relational triplet extraction (RTE) task, we provide an evaluation
example in Table 9. Given the urban text sentence ”Columbia University is a
private Ivy league research university in New York City.”, the human
annotators are required to first label the triplet contained in this urban
text, described as triplet _¡Columbia University, locate-in, New York City¿_.
Based on the label, then, they are instructed to evaluate how many true
triplets and false triplets in the results from the models. Finally, they will
fill out the evaluation form (i.e., the number of the true triplets and the
number of the false triplets). We will calculate the accuracy of results based
on these annotated forms.

For the knowledge graph completion (KGC) task, given the head entity
_¡Columbia University, Point (40.8075, 73.9626)¿_ and the tail entity _¡Empire
State Building, Point (40.7484, 73.9857)¿_ , the human annotators are first
required to complete their missing geospatial relationship from the five
relation candidate (i.e., DC, EC, PO, EQ and IN). Specifically, the annotation
could be achieved by manually visualizing the location of two entities given
on the map and following the RCC relation rule in Figure 6 to determine their
geographical relationships. Finally, they will fill out the evaluation results
(i.e., True/False). We will calculate the accuracy based on these evaluation
results.

###  D.2. GPT Evaluation Process

For the relational triplet extraction (RTE) task, we provide an evaluation
example in Table 9. Given the urban text sentence ”Columbia University is a
private Ivy league research university in New York City.”, and the model
extraction results (i.e., _¡Columbia University, locate-in, New York City¿_),
we directly prompt GPT-4 to fill out the evaluation form (i.e., the number of
the true triplets and the number of the false triplets). Then, the accuracy
could be obtained based on these self-evaluated results.

For the knowledge graph completion (KGC) task, given the head entity
_¡Columbia University, Point (40.8075, 73.9626)¿_ and the tail entity _¡Empire
State Building, Point (40.7484, 73.9857)¿_ and completed geospatial
relationship, we directly prompt GPT-4 the justify if the results are correct.
Specifically, we will first explicitly call eight external geospatial tools in
Table 8, and combine all these eight calculation results into the prompt. Then
we feed these prompts into GPT-4 to help it make a comparable accurate
evaluation. Finally, we obtain the evaluation results (i.e., True/False) for
every record. We will calculate the accuracy based on these evaluation
results.

However, such an evaluation process carries risks as we cannot guarantee the
infallibility of GPT-4. To ensure the validity of the evaluation, we have
conducted extensive experiments to demonstrate that the GPT evaluation method
aligns well with human evaluation.

###  D.3. Evaluation Consistency

We conduct experiments to assess the consistency between GPT-4’s evaluation
results and human evaluation results. In this work, we follow GPTScore (Fu et
al., 2023a) to use Spearman coefficient (Zar, 2005) to investigate the
correlation between GPT evaluation and human evaluation. Based on the
evaluation results in Tabel 3, we can conduct correlation analysis separately
by the types of model and paradigm.

Comparison of LLM Backbones. As shown in Figure 10, we observe that the
Spearman correlation coefficients on all LLM backbones are greater than $0.8$,
which means the human evaluation and GPT evaluation on these five LLM
backbones is highly correlated.

Comparison of LLM Paradigm. Moreover, we also conduct correlation under
different LLM paradigms for UrbanKGC tasks. As can be seen in Figure 11, all
four types of paradigm have a higher Spearman value above 0.85. Therefore,
deriving the GPT evaluation method is also applicable to different LLM
paradigms when finishing UrbanKGC tasks as its evaluation aligns well with
that of the human being.

Analysis of Evaluation Repeat. Nevertheless, the above correlation analysis is
based on results in Table 3, which are obtained after only single evaluation.
To deeply understand the evaluation mechanism, We repeatedly instruct GPT-4 to
generate 1, 3, 5, and 10 evaluation results to understand the variance of the
evaluation method. To save the cost, we perform the above analysis using
UrbanKGent-7B only on the NYC dataset. The results are shown in Table 10,
although the coefficient value will decrease as the number of repeated
experiments increases. However its value has always remained within a high
correlation range (roughly 0.85).

Table 10. The average Spearman correlation value between human evaluations and
GPT-4 evaluations. ”Repeat X times” refers to instructing GPT-4 to generate
judgments X times, and adopting the answer that appears most frequently (e.g.,
True/False for the KGC task and Number of the true triplet for the RTE task)
as the final decision.

Task | Repeat Once | Repeat 3 times | Repeat 5 times | Repeat 10 times
---|---|---|---|---
RTE | 98.56% | 87.31% | 86.42% | 85.27%
KGC | 94.86% | 91.23% | 90.09% | 91.24%

##  Appendix E Case Study

As shown in Figure 8, we present the detailed reasoning process of constructed
UrbanKGent-13B when finishing urban relational triplet extraction and
knowledge graph completion task. Since the iterative self-refinement process
contains excessive text, we display the reasoning process of only one
iteration.

Figure 10. The Spearman correlation between the GPT evaluation and human’s
evaluation under five different LLM backbones (i.e., Llama-2-7B, Llama-2-13B,
Llama-2-70B, GPT-3.5 and GPT-4).  Figure 11. The Spearman correlation between
the GPT evaluation and human’s evaluation under four LLM paradigms (i.e.,
Zero-shot learning (ZSL), In-context learning (ICL), Vanilla fine-tuning (VFL)
and UrbanKGent inference).

##  Appendix F Prototype System

We have deployed the constructed UrbanKGent-13B as a prototype system to offer
urban knowledge graph construction service. The project service could be found
in
Github111111https://htmlpreview.github.io/?https://raw.githubusercontent.com/usail-
hkust/UrbanKGent/main/UrbanKGent%20Demo/index.html. Figure 12 shows the
screenshot of our demo system. Specifically, for the user, the system provides
UrbanKGC service including relational triplet extraction and knowledge graph
completion task. Following the instruction template proposed in this paper,
the user can input urban text in the web shown in Figure 12 and the web will
return extracted triplet. The user can also input two entities name and their
geometry value, and the web serive will return completed geospatial
relationship between them.

##  Appendix G Evaluation on Larger Dataset

As shown in Table 6, we derive UrbanKGent-13B for urban knowledge graph
construction using constructed large-scale dataset in NYC and CHI.
Specifically, we directly use the text record in NYC-Large and CHI-Large for
the relational triplet extraction task. Then, we randomly sample the head-tail
entity pairs (both of head and tail entities contain geometry information)
from these triplets for knowledge graph completion. Nevertheless, iterating
all head-tail entity pairs is a time-consuming task, so we just construct a
KGC dataset consistent with the sacle of the RTE dataset. By performing RTE
and KGC tasks, we obtain two large-scale UrbanKGs shown in Table 6. Compared
with existing dataset and benchmark UUKG (Ning et al., 2023), we can clearly
observe that our agent can only use one-fifth data to construct the UrbanKG
with the same scale entities and triplets, but extend the relationship types
to a thousand times. It is worth noting that all construction process is
completed by a LLM agent without any mannul effort. We think it is the core of
this work.

Figure 12. The prototype system of UrbanKGent.

Moreover, we also report the performance of the RTE and KGC in NYC-Large
dataset. To save the cost of GPT self-evaluation, we only choose the best
approaches in ZSL, ICL, VFT and UrbanKGent Inference. The experimental results
are shown in Table 11. As can be seen, the fine-tuned UrbanKGC agent, whether
7B or 13B version, could achieve state-of-art performance on UrbanKGC tasks.

Table 11. The experimental results of relational triplet extraction (RTE) and
knowledge graph completion (KGC) on the NYC-Large dataset. To save the cost,
We choose the best approaches in zero-shot-learning (ZSL), In-context-learning
(ICL), Vanilla Fine-tuning (VFT), and UrbanKGent Inference setting.

Models | GPT(acc/confidence) | Human(acc)
---|---|---
RTE | KGC | RTE | KGC
ZSL | 0.36/4.01 | 0.45/3.95 | 0.42 | 0\. 31
ICL | 0.39/4.36 | 0.48/4.15 | 0.48 | 0.39
VFT | 0.37/4.05 | 0.46/3.98 | 0.45 | 0.35
| UrbanKGent
---
Inference
0.43/4.13 | 0.51/3.87 | 0.51 | 0.43
UrbanKGent-7B | 0.44/4.27 | 0.50/4.07 | 0.53 | 0.44
UrbanKGent-13B | 0.46/4.56 | 0.52/3.67 | 0.55 | 0.46

##  Appendix H Complexity and Effiency Analysis

We make a comparison with the four paradigms to demonstrate the advantages of
the constructed agent, which is shown in Table 5. Compared with Zero-shot
reasoning (ZSL), In-context Learning (ICL), Vanilla Fine-tuning (VFT), and
UrbanKGent Inference, UrbanKGent can incorporate extra urban knowledge, invoke
external tools and iteratively self-refine to help better complete UrbanKGC
tasks.

Moreover, we also provide comprehensive efficiency analysis to show the
latency and cost of different models when completing UrbanKGC tasks.
Specifically, we report the total inference time and cost121212We subscribe to
the NVIDIA A800 computing resources and GPT service from HKUST(GZ). Following
the standard price instruction of HKUST(GZ), we could calculate cost of GPT-
based baselines or Llama-based baselines. But in order to protect privacy of
the university, we do not provide detailed prices information here. of each
method completing with 1,000 RTE and KGC tasks. For the cost of GPT-4 service,
we first count the number of prompt token and completion token spent on 1,000
RTE and KGC tasks, and then calculate the cost based on billing standards. As
for the cost of UrbanKGent-13B, we first count the GPU running time spent on
1,000 RTE and KGC tasks, and then calculate the cost based on the A800
charging standard provided by HKUST(GZ).