# On the Opportunities and Challenges of Foundation Models for Geospatial
Artificial Intelligence

Gengchen Mai<EMAIL_ADDRESS>0000-0002-7818-7309 SEAI Lab, Department
of Geography, University of GeorgiaAthensGeorgiaUSA30605 ,  Weiming Huang
<EMAIL_ADDRESS>0000-0002-3208-4208 School of Computer Science and
Engineering, Nanyang Technological UniversitySingapore ,  Jin Sun
<EMAIL_ADDRESS>School of Computing, University of GeorgiaAthensGeorgiaUSA ,
Suhang Song<EMAIL_ADDRESS>0000-0003-1934-632X College of Public Health,
University of GeorgiaAthensGeorgiaUSA ,  Deepak Mishra<EMAIL_ADDRESS>0000-0001-8192-7681 Department of Geography, University of
GeorgiaAthensGeorgiaUSA ,  Ninghao Liu<EMAIL_ADDRESS>0000-0002-9170-2424 School of Computing, University of GeorgiaAthensGeorgiaUSA
,  Song Gao<EMAIL_ADDRESS>0000-0003-4359-6302 Geospatial Data Science
Lab, Department of Geography, University of Wisconsin-
MadisonMadisonWisconsinUSA ,  Tianming Liu<EMAIL_ADDRESS>0000-0002-8132-9048
School of Computing, University of GeorgiaAthensGeorgiaUSA ,  Gao Cong
<EMAIL_ADDRESS>0000-0002-4430-6373 School of Computer Science and
Engineering, Nanyang Technological UniversitySingapore ,  Yingjie Hu
<EMAIL_ADDRESS>0000-0002-5515-4125 GeoAI Lab, Department of Geography,
University at BuffaloBuffaloNew YorkUSA ,  Chris Cundy<EMAIL_ADDRESS>0000-0002-4608-4110 Department of Computer Science, Stanford
UniversityStanfordCaliforniaUSA ,  Ziyuan Li<EMAIL_ADDRESS>0009-0001-1835-3336 School of Business, University of
ConnecticutStorrsConnecticutUSA ,  Rui Zhu<EMAIL_ADDRESS>0000-0002-8910-9445 School of Geographical Sciences, University of
BristolBristolUnited Kingdom and  Ni Lao<EMAIL_ADDRESS>0000-0002-4034-7784
GoogleMountain ViewCaliforniaUSA

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Large pre-trained models, also known as _foundation models_ (FMs), are trained
in a task-agnostic manner on large-scale data and can be adapted to a wide
range of downstream tasks by fine-tuning, few-shot, or even zero-shot
learning. Despite their successes in language and vision tasks, we have yet
seen an attempt to develop foundation models for geospatial artificial
intelligence (GeoAI). In this work, we explore the promises and challenges of
developing multimodal foundation models for GeoAI. We first investigate the
potential of many existing FMs by testing their performances on seven tasks
across multiple geospatial subdomains including Geospatial Semantics, Health
Geography, Urban Geography, and Remote Sensing. Our results indicate that on
several geospatial tasks that only involve text modality such as toponym
recognition, location description recognition, and US state-level/county-level
dementia time series forecasting, these task-agnostic LLMs can outperform
task-specific fully-supervised models in a zero-shot or few-shot learning
setting. However, on other geospatial tasks, especially tasks that involve
multiple data modalities (e.g., POI-based urban function classification,
street view image-based urban noise intensity classification, and remote
sensing image scene classification), existing foundation models still
underperform task-specific models. Based on these observations, we propose
that one of the major challenges of developing a FM for GeoAI is to address
the multimodality nature of geospatial tasks. After discussing the distinct
challenges of each geospatial data modality, we suggest the possibility of a
multimodal foundation model which can reason over various types of geospatial
data through geospatial alignments. We conclude this paper by discussing the
unique risks and challenges to develop such a model for GeoAI.

Foundation Models, Geospatial Artificial Intelligence, Multimodal Learning

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Natural
language processing††ccs: Computing methodologies Unsupervised learning††ccs:
Applied computing Earth and atmospheric sciences††ccs: Computing methodologies
Computer vision††ccs: Computing methodologies Knowledge representation and
reasoning††ccs: Computing methodologies Neural networks

##  1\. Introduction

Recent trends in machine learning (ML) and artificial intelligence (AI) speak
to the unbridled powers of data and computing. Extremely large models trained
on Internet-scale datasets have achieved state-of-the-art (SOTA) performance
on a diverse range of learning tasks. In particular, their unprecedented
success has spurred a _paradigm shift_ in the way that modernday ML models are
trained. Rather than learning task-specific models from scratch (Gritta et
al., 2018; Wang et al., 2020; Lam et al., 2018), such pre-trained models (so-
called “foundation models (FMs)” (Bommasani et al., 2021)) are _adapted_ via
fine-tuning or few-shot/zero-shot learning strategies and subsequently
deployed on a wide range of domains (Brown et al., 2020; Radford et al.,
2021a). Such FMs allow for the transfer and sharing of knowledge across
domains, and mitigate the need for task-specific training data. Examples of
foundation models are 1) large language models (${LLM}$) such as PaLM (Wei et
al., 2022), LLAMA (Touvron et al., 2023), GPT-3 (Brown et al., 2020),
InstrucGPT (Ouyang et al., 2022), and ChatGPT (OpenAI, 2022); 2) large vision
foundation models such as Imagen (Saharia et al., 2022), Stable Diffusion
(Rombach et al., 2022), DALL$\cdot$E2 (Ramesh et al., 2022), and SAM (Kirillov
et al., 2023); 3) large multimodal foundation models such as CLIP (Radford et
al., 2021a), OpenCLIP (Ilharco et al., 2021), BLIP (Li et al., 2022),
OpenFlamingo (Awadalla et al., 2023), KOSMOS-1 (Huang et al., 2023a), and
GPT-4 (OpenAI, 2023); and 4) large reinforcement learning foundation models
such as Gato (Reed et al., 2022).

Despite their successes, there exists very little work exploring the
development of an analogous foundational model for geospatial artificial
intelligence (GeoAI), which lies at the intersection of geospatial scientific
discoveries and AI technologies (Janowicz et al., 2020; Goodchild and Li,
2021; Mai et al., 2022b). The key technical challenge here is the inherently
_multimodal_ nature of GeoAI. The core data modalities in GeoAI include text,
images (e.g., remote sensing or street view images), trajectory data,
knowledge graphs, and geospatial vector data (e.g., map layers from
OpenStreetMap), all of which contain important geospatial information (e.g.,
geometric and semantic information). Each modality exhibits special structures
that require its own unique representation – effectively combining all these
representations with appropriate inductive biases in a single model requires
careful design. The _multimodal_ nature of GeoAI hinders a straightforward
application of existing pre-trained FMs across all GeoAI tasks.

In this paper, we lay the groundwork for developing FMs for GeoAI. We begin by
providing a brief overview of existing foundation models in Section 2. Then in
Section 3, we investigate the potential of existing FMs for GeoAI by
systematically comparing the performances of several popular foundation models
with many state-of-the-art fully supervised task-specific machine learning or
deep learning models on various tasks from different geospatial domains: 1)
Geospatial Semantics: toponym recognition and location description recognition
task; 2) Health Geography: US state-level and county-level dementia death
count time series forecasting task; 3) Urban Geography: Point-of-interest
(POI) based urban function classification task and street-level image-based
noise intensity classification task; 4) Remote Sensing: remote sensing (RS)
image scene classification task. The advantages and problems of FM on
different geospatial tasks are discussed accordingly. Next, in Section 4, we
detail the challenges involved in developing FMs for GeoAI. Creating one
single FM for all GeoAI data modalities can be a daunting task. To address
this, we start this discussion by examining each data modality used in GeoAI
tasks. Then, we propose our vision for a novel multimodal FM framework for
GeoAI that tackles the aforementioned challenges. Finally, we highlight some
potential risks and challenges that should be considered when developing such
general-purpose models for GeoAI in Section 5 and conclude this paper in
Section 6.

Our contributions can be summarized as follows:

  * •

To the best of our knowledge, this is the first work that systematically
examines the effectiveness and problems of various existing cutting-edge
foundation models on different geospatial tasks across multiple geoscience
domains111This work is a significant extension of our previous 4-page vision
paper published in ACM SIGPATIAL 2022 (Mai et al., 2022a) by adding five
additional tasks in Health Geography, Urban Geography, and Remote Sensing
domains.. We establish various FM baselines on seven geospatial tasks for
future Geospatial Artificial General Intelligence (GeoAGI) research.

  * •

We discuss the challenges of developing a multimodal foundation model for
GeoAI and provide a promising framework to achieve this goal.

  * •

We discuss the risks and challenges that need to be taken into account during
the development and evaluation process of the multimodal geo-foundation model.

##  2\. Related Work

###  2.1. Language Foundation Model

In less than a decade, computational natural language capabilities have been
completely revolutionized (Peters et al., 2018; Kenton and Toutanova, 2019;
Raffel et al., 2020; Brown et al., 2020) by large-scale language modeling
(LLMs). Language modeling (Jurafsky and Martin, 2009) is the simple task of
predicting the next token in a sequence given previous tokens222There is also
a different variant which predicts masked spans in text (Kenton and Toutanova,
2019; Raffel et al., 2020)., and it corresponds to a self-supervised objective
in the sense that no human labeling is needed besides a natural text corpus.
When applied to vast corpora such as documents of diverse topics from the
internet, LLMs gain significant language understanding and generation
capabilities. Various transfer-learning and scaling studies (Kaplan et al.,
2020; Hernandez et al., 2021; Hoffmann et al., 2022) have demonstrated an
almost linear relationship between downstream task quality and the log sizes
of self-supervised model and data. Combined with the ever-increasing
availability of data and computing, language modeling has become a reliable
approach for developing increasingly powerful models.

Representative examples of these LLMs are the OpenAI GPTs (Radford et al.,
2018, 2019; Brown et al., 2020; OpenAI, 2023). By pretraining from vast
amounts of Web data, the GPT models gain knowledge of almost all domains on
the Web, which can be leveraged to solve problems of diverse verticals (Brown
et al., 2020). The interfaces to access such knowledge have become
increasingly simple and intuitive – ranging from supervised fine-tuning with
labeled data (Radford et al., 2018, 2019), to few-shot learning (Brown et al.,
2020) and instructions (Ouyang et al., 2022), to conversation (OpenAI, 2022)
and multimodality (OpenAI, 2023). In this study, we provide a comprehensive
analysis of the potentials and limitations of GPT and other LLMs when applied
to different geospatial domains.

###  2.2. Vision Foundation Model

Computer vision has long been dominated by task-specific models: for example,
YOLO (Redmon et al., 2016) for object detection, Detectron (Wu et al., 2019a)
for instance segmentation, and SRGAN (Ledig et al., 2017) for image super-
resolution. The newest example is Meta AI’s Segment Anything Model (SAM)
(Kirillov et al., 2023), which is designed for interactive object
segmentation. ResNet (He et al., 2016a) trained on ImageNet (Deng et al.,
2009) has been used as the backbone feature extractor for many such tasks. It
can be seen as the early form of a vision foundation model.

Inspired by the great success of language foundation models, the computer
vision community builds large-scale vision foundation models that can be
adapted to any vision task. The most direct adoption of the idea from language
models in computer vision is the image generation models. Since the dominance
of Generative Adversarial Networks (Goodfellow et al., 2020; Karras et al.,
2019), the quality of image generation models has seen a major breakthrough
via the development of diffusion-based models (Ho et al., 2020). Imagen
(Saharia et al., 2022) builds on large transformer-based language models to
understand text prompts and generates high-fidelity images using diffusion
models. DALL-E$\cdot$2 (Ramesh et al., 2022) trains a diffusion decoder to
invert an image encoder from visual-language models such as CLIP. After pre-
training, it is able to generate images of various styles and characteristics.
Stable Diffusion (Rombach et al., 2022) uses a Variational Autoencoder (VAE)
(Kingma and Welling, 2014) to convert raw images from pixel space to latent
space where the diffusion processes are more manageable and stable. It has
shown great flexibility in conditioning over text, pose, edge maps, semantic
maps, and scene depths (Zhang and Agrawala, 2023). GigaGAN (Kang et al.,
2023), on the other hand, is a recent attempt of scaling up GAN models.

Vision-Transformer (ViT) (Dosovitskiy et al., 2021) is a widely used
architecture in vision foundation models. Large-scale ViT has been developed
to scale up the model (Zhai et al., 2022). The Swin Transformer (Liu et al.,
2021) model is designed to handle the unique challenges of adapting regular
transformer models with various spatial resolutions in images. Other large-
scale non-transformer models are also developed to reach the same level of
performance: ConvNext (Liu et al., 2022b) is the “modernized” version of
convolutional neural networks that has a large number of parameters and shows
a similar level of performance as Swin Transformers. MLP-mixer (Tolstikhin et
al., 2021) is an architecture that utilizes only multi-layer perceptrons on
image data. It shows competitive scores on image classification datasets.

###  2.3. Multimodal Foundation Model

Developing artificial intelligence models that are capable of performing
multimodal reasoning and understanding on complex data is a promising idea.
Humans naturally perform multimodal reasoning in daily life (Pearson, 2019)
for example, when a person is thinking about the concept of ‘dog’, they will
not only think about the English word and its meaning but also a visual image
and a sound associated with it. In the context of geospatial tasks, multimodal
data are ubiquitous. In general, data from different modalities provide
different ‘views’ that complement each other and provide more information to
facilitate a holistic understanding of the data.

Recently, much progress has been made in building large-scale multimodal
foundation models for joint reasoning from various domains, in particular,
vision and language. CLIP (Radford et al., 2021a; Ilharco et al., 2021) is one
of the first widely-adopted vision-language joint training frameworks. It uses
self-supervised contrastive learning to learn a joint embedding of visual and
text features. BLIP (Li et al., 2022) improves over CLIP by training on
synthetically-generated captions from internet-collected images. It is
designed to handle both visual-language understanding and generation tasks.
BEiT-3 (Wang et al., 2022) is a general-purpose multimodal foundation model
that achieves state-of-the-art performance on both vision and vision-language
tasks. It combines features from multi-modality expert networks. Florence
(Yuan et al., 2021) is a vision-language foundation model that learns
universal visual-language representations for objects, scenes, images, videos,
as well as captions. Similarly, KOSMOS-1 (Huang et al., 2023a) learns from
web-scale multimodal data including text and image pairs. It can transfer
knowledge from one modality to another. Flamingo (Alayrac et al., 2022) is a
family of visual language models that can be adapted to novel tasks using only
a few annotated examples, i.e., few-shot learning. It encodes images or videos
as inputs along with textual tokens to jointly reason about vision tasks. The
newest version of the GPT model, the GPT-4 (OpenAI, 2023), also claims to
perform multimodal analysis including text, audio, images, and videos.

##  3\. Exploration of the Effectiveness of Existing FMs on Various Geospatial
Domains

The first question we would like to ask is how the existing cutting-edge
foundation models perform when compared with the state-of-the-art fully
supervised task-specific models on various geospatial tasks. Geography is a
very broad discipline that includes various subdomains such as Geospatial
Semantics (Kuhn, 2005; Jones and Purves, 2008; Janowicz et al., 2012; Hu,
2018; Mai et al., 2021), Health Geography (Kearns and Moon, 2002; Rosenberg,
2014; Chang et al., 2021), Urban Geography, (Zhang et al., 2018b; Cai et al.,
2020; Zhu et al., 2020; Kang et al., 2021; Huang et al., 2023c), Remote
Sensing (Mishra et al., 2012; Burke et al., 2021; Rolf et al., 2021; Lee et
al., 2021a; Elmustafa et al., 2022), and so on. To address the aforementioned
question, in the following, we conduct experiments using various FMs on
different tasks in the four geospatial subdomains mentioned earlier. The
advantages and weaknesses of existing FMs will be discussed in detail.

###  3.1. Geospatial Semantics

⬇

[Instruction]...

Paragraph: Alabama State Troopers say a Greenville man has died of his
injuries after being hit by a pickup truck on Interstate 65 in Lowndes County.

Q: Which words in this paragraph represent named places?

A: Alabama; Greenville; Lowndes

...

\--

Paragraph: The Town of Washington is to what Williamsburg is to Virginia.

Q: Which words in this paragraph represent named places?

A:Washington; Williamsburg; Virginia

Listing 1: Typonym recognition with LLMs, e.g., GPT-3. Yellow block: the text
snippet to be annotated. Orange box: GPT-3 outputs. 8 few-shot samples are
used in this prompt. We only show 1 here while skipping others with "…" to
save space.

⬇

[Instruction]...

Paragraph: Papa stranded in home. Water rising above waist. HELP 8111 Woodlyn
Rd, 77028 #houstonflood

Q: Which words in this paragraph represent location descriptions?

A: 8111 Woodlyn Rd, 77028

...

\--

Paragraph: HurricaneHarvey Help Need AT 7506 Jackrabbit Rd, Houston, TX 77095.

Q: Which words in this paragraph represent location descriptions?

A:7506 Jackrabbit Rd, Houston, TX 77095

Listing 2: Location description recognition with LLMs, e.g., GPT-3. Yellow
block: the input text snippet. Orange box: GPT-3 outputs. 11 few-shot samples
are used while 1 is shown.

As a starting point for our discussion, we first demonstrate empirically the
promise of leveraging LLMs for solving geospatial semantics tasks. We hope
that our results not only demonstrate the effectiveness of such general-
purpose, few-shot learners in the geospatial semantics domain, but also
challenges the current paradigm of training task-specific models as a common
practice in GeoAI research.

We compare the performance of 4 pre-trained GPT-2 (Radford et al., 2019)
models of varying sizes provided by Huggingface as well as the most recent
GPT-3 (Brown et al., 2020) (i.e., text-davinci-002), InstructGPT (Ouyang et
al., 2022) (i.e., text-davinci-003), and ChatGPT (OpenAI, 2022) (i.e.,
gpt-3.5-turbo) models developed by OpenAI with multiple _supervised, task-
specific_ baselines on two representative geospatial semantics tasks: (1)
toponym recognition (Gritta et al., 2018; Wang and Hu, 2019), and (2) location
description recognition (Hu et al., 2020).

####  3.1.1. Toponym Recognition

Toponym recognition is a subtask of named entity recognition (NER), with the
goal of identifying named places from a given text snippet. We use the Hu2014
(Hu et al., 2014) and Ju2016 (Ju et al., 2016) benchmark datasets as two
representative datasets for this task. We adapt 7 pre-trained GPT models to
perform toponym recognition tasks by using appropriate prompts containing few-
shot training examples. In the prompt, we provide several training samples as
few-shot learning samples in the form of natural language instructions. One
example of such a prompt is illustrated in Listing LABEL:ls:prompt-tr, while
the full prompts can be found in List LABEL:ls:prompt-tr-all in Appendix A.1.
It is worth noting that ChatGPT , as a foundation model, is optimized for
chatbot purposes and expects conversational inputs rather than a single big
prompt. In order to conduct a controlled experiment, we first use the same
prompt shown in Listing LABEL:ls:prompt-tr to instruct all 7 pre-trained GPT
models to perform toponym recognition. We also convert the few-shot examples
into a list of conversations and use them as the inputs for ChatGPT which is
denoted as ChatGPT (Con.) while the ChatGPT using the original prompt is
indicated as ChatGPT (Raw.).

Table 1 compares all 8 GPT models with 15 baselines on two datasets – Hu2014
(Hu et al., 2014) and Ju2016 (Ju et al., 2016). The same test sets have been
used to evaluate the performances of all models. Please refer to Hu et al. (Hu
et al., 2014) and Ju et al. (Ju et al., 2016) for detailed descriptions of
both datasets. Those 15 baselines are classified into three groups as shown in
Table 1: (A) general NER (named entity recognition) models; (B) no neural
network (NN) based geoparsers; (C) fully supervised task-specific NN-based
geoparsers. All models in Group C are trained in a supervised manner on the
same separated training datasets. With the exception of the smallest GPT2
model, all other LLMs consistently outperform the fully-supervised baselines
on the Hu2014 dataset, even though they only require a small set of natural
language instructions without any additional training. GPT-3 in particular
demonstrated an 8.7% performance improvement over the previous SOTA
(TopoCluster (DeLozier et al., 2016)). Interestingly, new GPT models such as
InstructGPT and ChatGPT do not show higher performances on the Hu2014 dataset.
While InstructGPT shows a smaller performance drop which is acceptable, two
ChatGPT models show more significant performance decreases. One reasonable
hypothesis is that ChatGPT is further optimized based on InstructGPT for
chatbot applications that may not be “flexible” enough to be adapted to new
tasks such as toponym recognition.

Based on previous studies (Wang and Hu, 2019; Wang et al., 2020), the Ju2016
dataset is a very difficult task. On this dataset, we found that GPT2-XL
outperforms the previous SOTA (NeuroTPR (Wang et al., 2020)) by over 2.5%
while using only _8 few-shot examples in the prompt_. In contrast, a task-
specific model, such as NeuroTPR, requires supervised training on 599 labeled
tweets and labeled sentences generated from 3000 Wikipedia articles. GPT-3 and
InstructGPT does not show performance improvement on the Ju2016 dataset over
GPT2-XL. Similar to the finding on the Hu2014 dataset, ChatGPT shows a
significant performance decrease on the Ju2016 dataset. In accordance with
existing empirical findings (Radford et al., 2019; Brown et al., 2020), we
also found that the performance of these LLMs tended to scale with the number
of learnable parameters.

####  3.1.2. Location Description Recognition

The location description recognition task is slightly more challenging – given
a text snippet (e.g., tweets), the goal is to recognize more fine-grained
location descriptions such as home addresses, highways, roads, and
administration regions. HaveyTweet2017 (Hu et al., 2020) is used as one
representative benchmark dataset for this task. The same set of pre-trained
GPT models and 15 baselines are used for this task. Listing LABEL:ls:prompt-
ldr shows one example prompt used in this task and the full prompt can be seen
in Listing LABEL:ls:prompt-ldr-all in Appendix A.1.

Table 1 summarizes the evaluation results of different models on the
HaveyTweet2017 dataset. The same test set of HaveyTweet2017 is used to
evaluate all GPT models as well as 15 baseline models. On the HaveyTweet2017
dataset, GPT-3 achieves the best recall score across all methods. However, all
LLMs have rather low precision (and therefore low F1-scores). This is because
LLMs implicitly convert the location description recognition problem into a
natural language generation problem (see List LABEL:ls:prompt-ldr), meaning
that they are not guaranteed to generate tokens that appear in the input text.
Based on the experimental results in Table 1, we can clearly see that by using
just a small number of few-shot samples, LLMs can outperform the fully-
supervised, task-specific models on well-defined geospatial semantics tasks.
This showcases the potential of LLMs to dramatically reduce the need for
customized architectures or large labeled datasets for geospatial tasks.
However, how to develop appropriate prompts to instruct LLMs for a given
geospatial semantics task require further investigation.

Table 1.  Evaluation results of various GPT models and baselines on two geospatial semantics tasks: (1) toponym recognition (Hu2014 (Hu et al., 2014) and Ju2016 (Ju et al., 2016)) and (2) location description recognition (HaveyTweet2017 (Hu et al., 2020)) . We classify all models into four groups: (A) General NER; (B) No Neural Network (NN) based geoparsers; (C) Fully-supervised NN-based geoparsers; (D) Few-show learning with LLMs. "(#Param)" indicates the number of learnable parameters of LLMs. "(nar. loc.)" and "(bor. loc.)" indicate narrow location models and broad location models defined in (Wang et al., 2020). The results of all baselines (i.e., models in Group A, B, and C) are obtained from (Wang and Hu, 2019) and (Wang et al., 2020) except "0.675†", which is obtained by rerunning the official code of (Wang et al., 2020). The evaluation results of different GPT models (Group D) are done by using pre-trained GPT2/GPT-3/InstructGPT/ChatGPT models with appropriate prompts. The results of four GPT2 models are obtained by using Huggingface pre-trained GPT2models with various model sizes. The last four models are obtained by using various OpenAI’s GPT models – text-davinci-002, text-davinci-003, and gpt-3.5-turbo – which are denoted as GPT-3, InstructGPT, and ChatGPT respectively. Since ChatGPT expects conversational inputs rather than a single big prompt, we experiment with two versions of ChatGPT. ChatGPT (Raw.) indicates we use the same prompt as other GPT models while ChatGPT (Con.) indicates we convert the few-shot examples in the prompt into a list of conversations. ∗Due to OpenAI API limitations, we evaluate GPT-3, InstructGPT, and ChatGPT on randomly sampled 544 Ju2016 examples (10% of the dataset).  | Model | #Param | Toponym Recognition | Location Description Recognition
---|---|---|---|---
Hu2014 | Ju2016 | HaveyTweet2017
Accuracy $\downarrow$ | Accuracy $\downarrow$ | Precision $\downarrow$ | Recall $\downarrow$ | F-Score $\downarrow$
(A) | Stanford NER (nar. loc.) (Finkel et al., 2005) | - | 0.787 | 0.010 | 0.828 | 0.399 | 0.539
Stanford NER (bro. loc.) (Finkel et al., 2005) | - | - | 0.012 | 0.729 | 0.44 | 0.548
Retrained Stanford NER (Finkel et al., 2005) | - | - | 0.078 | 0.604 | 0.410 | 0.489
Caseless Stanford NER (nar. loc.) (Finkel et al., 2005) | - | - | 0.460 | 0.803 | 0.320 | 0.458
Caseless Stanford NER (bro. loc.) (Finkel et al., 2005) | - | - | 0.514 | 0.721 | 0.336 | 0.460
spaCy NER (nar. loc.) (Honnibal and Montani, 2017) | - | 0.681 | 0.000 | 0.575 | 0.024 | 0.046
spaCy NER (bro. loc.) (Honnibal and Montani, 2017) | - | - | 0.006 | 0.461 | 0.304 | 0.366
DBpedia Spotlight(Mendes et al., 2011) | - | 0.688 | 0.447 | - | - | -
(B) | Edinburgh (Alex et al., 2015) | - | 0.656 | 0.000 | - | - | -
CLAVIN (Wang and Hu, 2019) | - | 0.650 | 0.000 | - | - | -
TopoCluster (DeLozier et al., 2016) | - | 0.794 | 0.158 | - | - | -
(C) | CamCoder (Gritta et al., 2018) | - | 0.637 | 0.004 | - | - | -
Basic BiLSTM+CRF (Lample et al., 2016) | - | - | 0.595 | 0.703 | 0.600 | 0.649
DM NLP (top. rec.) (Wang et al., 2019) | - | - | 0.723 | 0.729 | 0.680 | 0.703
NeuroTPR (Wang et al., 2020) | - | 0.675† | 0.821 | 0.787 | 0.678 | 0.728
(D) | GPT2 (Radford et al., 2019) | 117M | 0.556 | 0.650 | 0.540 | 0.413 | 0.468
GPT2-Medium (Radford et al., 2019) | 345M | 0.806 | 0.802 | 0.529 | 0.503 | 0.515
GPT2-Large (Radford et al., 2019) | 774M | 0.813 | 0.779 | 0.598 | 0.458 | 0.518
GPT2-XL (Radford et al., 2019) | 1558M | 0.869 | 0.846 | 0.492 | 0.470 | 0.481
GPT-3 (Brown et al., 2020) | 175B | 0.881 | 0.811∗ | 0.603 | 0.724 | 0.658
InstructGPT (Ouyang et al., 2022) | 175B | 0.863 | 0.817∗ | 0.567 | 0.688 | 0.622
ChatGPT (Raw.) (OpenAI, 2022) | 176B | 0.800 | 0.696∗ | 0.516 | 0.654 | 0.577
ChatGPT (Con.) (OpenAI, 2022) | 176B | 0.806 | 0.656∗ | 0.548 | 0.665 | 0.601

###  3.2. Health Geography

The next set of experiments focuses on an important health geography problem –
dementia death counts time series forecasting for a given geographic region
such as cities, counties, states, etc. With a growing share of older adults in
the population, it is estimated that more than 7 million US adults aged 65 or
older were living with dementia in 2020, and the number could increase to over
9 million by 2030 and nearly 12 million by 2040 (Zissimopoulos et al., 2018).
Alzheimer’s disease, the most common type of dementia, has been reported to be
one of the top leading causes of death in the US, with 1 in 3 seniors dying
with Alzheimer’s or another dementia by 2019 (Association et al., 2022).
Notably, there are substantial and longstanding geographical disparities in
mortality due to dementia (Association et al., 2021; Akushevich et al., 2021).
Subnational planning and prioritizing dementia prevention strategies require
local mortality data. Prediction of dementia deaths at the sub-national level
will assist in informing future tailored health policies to eliminate
geographical disparities in dementia and to achieve national health goals.

In this work, we conduct time series forecasting on the number of deaths due
to dementia in two geographic region levels – state level and county level.
The dementia data are obtained from the US Centers for Disease Control and
Prevention Wide-ranging Online Data for Epidemiologic Research (CDC
WONDER333https://wonder.cdc.gov/ucd-icd10.html), which is a publicly available
dataset. Dementia deaths are classified according to the International
Classification of Diseases, Tenth Revision (ICD-10), including unspecified
dementia (F03), Alzheimer’s disease (G30), vascular dementia (F01), and other
degenerative diseases of nervous system, not elsewhere classified (G31)
(Kramarow and Tejada-Vera, 2019).

####  3.2.1. US State-Level Dementia Time Series Forecasting

We collect annual time series of dementia death counts for all 51 US states
between 1999 and 2020. The time series from 1999 to 2019 are used as training
data, and the numbers in 2020 are used as ground truth labels. The same set of
pre-trained GPT models used in Section 3.1 are utilized in this task.
Different from the geospatial semantics experiments, we utilize all GPT models
in a zero-shot setting since we think the historical time series data is
enough for a LLM to perform the forecasting. Listing LABEL:ls:prompt-alz-state
shows one example prompt we use in this experiment by using California as an
example.

With only 51 time series, each consisting of 22 data points, many sequential
deep learning models such as RNNs (recurrent neural networks) and Transformers
(Vaswani et al., 2017) are at risk of overfitting on this dataset. So we pick
the state-of-the-art machine learning-based time series forecasting model –
ARIMA (Autoregressive integrated moving average) as the fully supervised task-
specific baseline model. We train individual ARIMA models on each state’s time
series using data from 1999 to 2019, and perform forecasting on data in 2020.
Hyperparameter tuning is performed on all ARIMA hyperparameters to obtain the
best results. Additionally, we use persistence model (Notton and Voyant, 2018;
Paulescu et al., 2021) as a reference. A persistence model assumes that the
future value of a time series remains the same between the current time and
the forecast time. In our case, we use the dementia death count of each state
in 2019 as the prediction for the value in 2020.

Table 2 presents a comparison of model performances among different GPT models
and two baselines. Interestingly, all GPT2 models perform poorly on all
evaluation metrics. Their performances are even worse than the simple
persistence model. This suggests that GPT2 may struggle with zero-shot time
series forecasting. On the other hand, GPT-3, InstructGPT, and two ChatGPT
models demonstrate reasonable performances. Of particular interest is that
InstructGPT outperforms the best ARIMA model on all evaluation metrics even
though InstructGPT is not finetuned on this specific task. We propose two
hypothetical reasons for the strong performance of InstructGPT in the time
series forecasting task: 1) After training on a large-scale text corpus,
InstructGPT may have developed the intelligence necessary to perform zero-shot
time series forecasting, which is fundamentally an autoregressive problem. 2)
It is possible that InstructGPT and GPT-3 may be exposed to US state-level
dementia time series data during their training on the large-scale text
corpus.

While we cannot determine which of these reasons is the primary factor behind
InstructGPT’s success, these results are very encouraging. Similar to the
results in Table 1, two ChatGPT models underperform InstructGPT. More
experiment analysis can be seen in the county-level experiments.

Table 2.  Evaluation results of various GPT models and baselines on the US state-level dementia time series forecasting task. We classify all models into four groups: (A) Simple persistent model; (B) Fully supervised machine learning models such as ARIMA; (C) Zero-shot learning with LLMs. "(#Param)" indicates the number of learnable parameters of LLMs. The denotations of different GPT models are the same as Table 1. Four evaluation metrics are used: MSE (mean square error), MAE (mean absolute error), MAPE (mean absolute percentage error), and R2. $\uparrow$ and $\downarrow$ indicate the direction of better models for each metric. For all GPT models, we encode time series information between 1999 and 2019 in the prompt and let it generate data in 2020.  | Model | #Param | MSE $\downarrow$ | MAE $\downarrow$ | MAPE $\downarrow$ | R2 $\uparrow$
---|---|---|---|---|---|---
(A) Simple | Persistence (Notton and Voyant, 2018; Paulescu et al., 2021) | - | 985,179 | 630 | 0.096 | 0.971
(B) Supervised ML | ARIMA (Jenkins et al., 2011) | - | 562,768 | 462 | 0.067 | 0.984
(C) Zero shot LM | GPT2 (Radford et al., 2019) | 117M | 44,635,055 | 4,898 | 0.955 | -0.271
GPT2-Medium (Radford et al., 2019) | 345M | 42,315,630 | 4,616 | 0.745 | -0.209
GPT2-Large (Radford et al., 2019) | 774M | 39,039,733 | 4,250 | 0.779 | -0.132
GPT2-XL (Radford et al., 2019) | 1558M | 35,355,840 | 3,912 | 0.709 | -0.026
GPT-3 (Brown et al., 2020) | 175B | 587,263 | 474 | 0.070 | 0.983
InstructGPT (Ouyang et al., 2022) | 175B | 387,413 | 365 | 0.055 | 0.989
ChatGPT (Raw.) (OpenAI, 2022) | 176B | 1,143,675 | 623 | 0.121 | 0.967
ChatGPT (Con.) (OpenAI, 2022) | 176B | 4,224,811 | 1,131 | 0.240 | 0.890

⬇

[Instruction]This is a set of time series forecasting problems.

The ‘Paragraph‘ is a time series of the numbers of deaths from alzheimer’s
disease for one of US state from 1999 to 2019.

The goal is to predict the number of deaths from alzheimer’s disease at this
state in 2020. Please give a single number as the prediction.

\--

\--

Paragraph: At California, From 1999 to 2019, the numbers of deaths from
alzheimer’s disease are 6761 in 1999, 6760 in 2000, 7474 in 2001, 8366 in
2002, 9760 in 2003, 9806 in 2004, 11497 in 2005, 13520 in 2006, 13730 in 2007,
16395 in 2008, 16290 in 2009, 18000 in 2010, 19924 in 2011,20814 in 2012,
22061 in 2013, 22412 in 2014, 23606 in 2015, 24060 in 2016, 25017 in 2017,
25218 in 2018, and 25810 in 2019.

Q: Please forecast the number in 2020 at California?

A:26670

Listing 3: US state-level Alzimier time series forecasting with LLMs by zero-
shot learning. Yellow block: the historical time series data of one US state.
Orange box: the outputs of InstructGPT. Here, we use California as an example
and the correct answer is 29400.

⬇

[Instruction]This is a set of time series forecasting problems.

The ‘Paragraph‘ is a time series of the numbers of deaths from alzheimer’s
disease for one of US counties from 1999 to 2019.

The goal is to predict the number of deaths from alzheimer’s disease at this
county in 2020. Please give a single number as the prediction.

\--

\--

Paragraph: At Santa Barbara County, CA, from 1999 to 2019, the numbers of
deaths from alzheimer’s disease are 126 in 1999, 114 in 2000, 124 in 2001, 127
in 2002, 156 in 2003, 154 in 2004, 175 in2005, 172 in 2006, 171 in 2007, 248
in 2008, 204 in 2009, 241 in 2010, 260 in 2011, 297 in 2012, 283 in 2013, 308
in 2014, 358 in 2015, 365 in 2016, 334 in 2017, 363 in 2018, and 328 in 2019.

Q: Please forecast the number in 2020 at Santa Barbara County, CA?

A:345

Listing 4: US county-level Alzimier time series forecasting with LLMs by zero-
shot learning. Yellow block: the historical time series data of one US county.
Orange box: the outputs of InstructGPT. Here, we use Santa Barbara County, CA
as an example and the correct answer is 373.

####  3.2.2. US County-Level Dementia Time Series Forecasting

In terms of county-level data, we utilized the dementia death count time
series of all US counties with available data, resulting in a total of 2447 US
counties selected for analysis. We only considered counties with dementia
annual death records spanning more than four years between 1999 and 2020.
Similarly to Section 3.2.1, we utilize all available data up to the given year
for training ARIMA models and generating GPT prompts, and then make
predictions for the following year. We employ the same set of GPT models and
baselines as in the state-level experiment to conduct the county-level
experiment. Listing LABEL:ls:prompt-alz-county shows one example prompt we use
in this experiment by using Santa Barbara County, CA as an example.

Table 3.  Evaluation results of various GPT models and baselines on the US county-level dementia time series forecasting task. We use same model set and evaluation metrics as Table 2.  | Model | #Param | MSE $\downarrow$ | MAE $\downarrow$ | MAPE $\downarrow$ | R2 $\uparrow$
---|---|---|---|---|---|---
(A) Simple | Persistence (Notton and Voyant, 2018; Paulescu et al., 2021) | - | 1,648 | 16.9 | 0.189 | 0.979
(B) Supervised ML | ARIMA (Jenkins et al., 2011) | - | 1,133 | 15.1 | 0.193 | 0.986
(C) Zero shot LLMs | GPT2 (Radford et al., 2019) | 117M | 77,529 | 92.0 | 0.587 | -0.018
GPT2-Medium (Radford et al., 2019) | 345M | 226,259 | 108.1 | 0.611 | -2.824
GPT2-Large (Radford et al., 2019) | 774M | 211,881 | 94.3 | 0.581 | -1.706
GPT2-XL (Radford et al., 2019) | 1558M | 162,778 | 99.8 | 0.627 | -1.082
GPT-3 (Brown et al., 2020) | 175B | 1,105 | 14.5 | 0.180 | 0.986
InstructGPT (Ouyang et al., 2022) | 175B | 831 | 13.3 | 0.179 | 0.989
ChatGPT (Raw.) (OpenAI, 2022) | 176B | 4,115 | 23.2 | 0.217 | 0.955
ChatGPT (Con.) (OpenAI, 2022) | 176B | 3,402 | 20.7 | 0.231 | 0.944

Table 3 compares the results of different models. Similar findings can be seen
from these results. All GPT2 models perform poorly. However, both GPT-3 and
InstructGPT outperform the best ARIMA models on all evaluation metrics, while
two ChatGPT models underperform them. Among the two ChatGPT models, ChatGPT
(Con.) are slightly better than ChatGPT (Raw.) on all metrics except MAPE.

(a) Persistence

(b) ARIMA

(c) GPT2

(d) GPT2-Medium

(e) GPT2-Large

(f) GPT2-XL

(g) GPT-3

(h) InstructGPT

(i) ChatGPT (Raw.)

(j) ChatGPT (Con.)

Figure 1. Prediction error maps of each baseline and GPT model on US county-
level dementia death count time series forecasting task. The color on each US
count indicates the percentage error $PE=(Prediction-Label)/Label$ of each
model prediction on this county. Those counties in gray color indicate their
dementia data during 1999 and 2020 are not available.

To further understand the geographical distributions of prediction errors for
each model, we visualize the prediction errors of each model on each US county
in Figure 1. In the figure, the red color represents overestimations of the
corresponding model while the blue colors indicate underestimations. Moreover,
the intensity of the color indicates the magnitude of the prediction error,
with darker colors representing larger errors. We can see that Persistence,
ARIMA, GPT-3, and InstructGPT generally demonstrate better forecasting
performance. However, the prediction percentage errors are not uniformly
distributed across different US counties. As persistence uses the previous
year’s data as the prediction, Figure 1(a) indicates that the growth rates of
dementia death counts are uneven for different counties. The southwest of the
U.S. shows a recent increase in dementia death counts which leads the
persistence model to underestimate the true data. The current maps of
prediction errors show that the distributions of errors of GPT-3 and
InstructGPT are not uniform across the US counties, and it is unclear whether
the uneven distribution is due to the geographic bias encoded in the models or
the spatial heterogeneity of the growth rate of dementia death counts. Further
analysis is needed to determine the cause of these uneven distributions.

One obvious observation from Figure 1 is that all GPT2 models turn to
significantly underestimate the dementia data. To understand the cause of this
behavior and the superiority of GPT-3 and InstructGPT, we showcase the
generated answers of different GPT models for four US counties in Table 4.
From Table 4, it is evident that GPT2 in many times will repeat the
information provided in our prompt rather than generating novel predictions.
For example, in the Clarke County, GA and Santa Barbara County, CA cases, all
three GPT2 models (i.e., GPT2-Medium, GPT2-Large, and GPT2-XL) predict the
same numbers as the data in 1999. This suggests that these models rely heavily
on the prompt information instead of learning from the time series data, which
could explain their inferior performance compared to other models such as
GPT-3 and InstructGPT. In the other two counties’ cases, the predictions of
the GPT2 models vary significantly. In most cases, both InstructGPT and
ChatGPT (Raw.) generate a single number as the prediction, indicating that
they understand the task they are expected to perform. The only exception is
the Santa Barbara County case, where ChatGPT (Raw.) generates a short sentence
containing a reasonable prediction. However, based on our evaluation, the
predictions of ChatGPT (Raw.) are not as good as those of GPT-3.
Interestingly, when using ChatGPT in a conversational context, i.e., ChatGPT
(Con.), ChatGPT usually returns a very long sentence. In the New York County
case, ChatGPT (Con.) is unable to give a prediction, suggesting that ChatGPT
is useful in a chatbot context but may not be the best choice for other tasks
such as time series forecasting.

Table 4.  Generated answers by various GPTs models for four US counties in US county-level dementia time series forecasting problem. “County” and “Label” columns indicate the US county and its true dementia death count in 2020. “Generated Answer” indicates the generated answer for different GPT models in which the numbers in bold are the predictions we extract from these answers.  County | Label | Model |  Generated Answer
---|---|---|---
Clarke County, GA | 79 | GPT2 |  The number of deaths from alzheimer’s disease is 21 in 1999, 37 in 2000, 32 in 2001, 25 in 2002, 29 in 2003, 44 in 2004, 38 in 2005, 41 in 2006, 52 in 2007, 46 in 2008
GPT2-Medium |  The number of deaths from alzheimer’s disease in Clarke County, GA is 21 in 2020.
GPT2-Large |  The number of deaths from alzheimer’s disease in 2020 is: 21 in 2020, 37 in 2021, 32 in 2022, 25 in 2023, 29 in 2024, 44 in 2025, 38 in 2026, 42 in 2027, 42
GPT2-XL |  The number in 2020 is: 21.
GPT-3 |  The number of deaths from Alzheimer’s disease in 2020 at Clarke County, GA is forecast to be 81.
InstructGPT |  83
ChatGPT (Raw.) |  78
ChatGPT (Con.) |  Based on the available historical data, the forecast for the number of deaths from Alzheimer’s disease in Clarke County, GA in 2020 would be 81.
New York County, NY | 835 | GPT2 |  The number of deaths from alzheimer’s disease is 91 in 1999, 121 in 2000, 118 in 2001, 155 in 2002, 214 in 2003, 243 in 2004, 219 in 2005, 294 in 2006, 296 in 2007, 364 in 2008
GPT2-Medium |  The number of deaths from alzheimer’s disease in New York County, NY is 9,965.
GPT2-Large |  The number of deaths from alzheimer’s disease in New York County, NY is 91 in 2020.
GPT2-XL |  The number in 2020 is: 901.
GPT-3 |  The number of deaths from alzheimer’s disease in 2020 at New York County, NY is 932.
InstructGPT |  902
ChatGPT (Raw.) |  855
ChatGPT (Con.) |  Based on the historical data provided, a common approach for time series forecasting is to use methods such as exponential smoothing or ARIMA. However, without further information on the specific pattern or trend in the data, it is difficult to provide an accurate prediction for the number of deaths from alzheimer’s disease in New York County, NY for 2020. Therefore, I am unable to provide a single number as the prediction.
San Francisco County, CA | 592 | GPT2 |  The number of deaths from alzheimer’s disease is 1,000. The number of deaths from alzheimer’s disease is 2,000. The number of deaths from alzheimer’s disease is 3,000. The number of deaths from
GPT2-Medium |  The number of deaths from alzheimer’s disease is 201 in 2020.
GPT2-Large |  The number of deaths from alzheimer’s disease in 2020 is 574.
GPT2-XL |  The number in 2020 is 201.
GPT-3 |  571
InstructGPT |  590
ChatGPT (Raw.) |  579
ChatGPT (Con.) |  Based on the historical data provided, a possible forecasting method is to use a time series model, such as ARIMA or exponential smoothing. Without fitting such a model, a simple approach would be to assume a trend of similar magnitude to recent years and a similar level of annual variation. Using this approach, a possible prediction for the number of deaths from alzheimer’s disease in San Francisco County, CA in 2020 is 583.
Santa Barbara County, CA | 373 | GPT2 |  The number of deaths from alzheimer’s disease is 126 in 1999, 114 in 2000, 124 in 2001, 127 in 2002, 156 in 2003, 154 in 2004, 175 in 2005, 172 in 2006, 171 in 2007, 248 in 2008
GPT2-Medium |  The number of deaths from alzheimer’s disease in Santa Barbara County, CA is 126 in 2020.
GPT2-Large |  The number of deaths from alzheimer’s disease in Santa Barbara County, CA is: 126 in 2020.
GPT2-XL |  The number in 2020 is: 126.
GPT-3 |  The number of deaths from alzheimer’s disease in 2020 at Santa Barbara County, CA is expected to be about 350.
InstructGPT |  345
ChatGPT (Raw.) |  I predict the number of deaths from alzheimer’s disease in 2020 at Santa Barbara County, CA to be 356.
ChatGPT (Con.) |  Based on the historical data provided, the prediction for the number of deaths from Alzheimer’s disease in 2020 at Santa Barbara County, CA is 327.

###  3.3. Urban Geography

The third set of FM experiments focuses on research problems in the Urban
Geography domain. Two representative tasks are selected: 1) an urban function
task that aims at predicting the urban functions of a geographic region based
on the Points of Interest (POIs) within it (Yao et al., 2017; Zhai et al.,
2019; Niu and Silva, 2021; Huang et al., 2022, 2023c); 2) an urban perception
task that focuses on predicting the urban neighborhood characteristics (e.g.,
housing price, safety, noise intensity level) based on street view imagery
(SVI) (Zhang et al., 2018b; Kang et al., 2021; Zhao et al., 2023a). Since
these tasks involve different data modalities such as point data, text, and
images, we use different foundation models to handle each task.

####  3.3.1. POI-Based Urban Function Classification

The first experiment focuses on predicting the urban functions of a geographic
region based on the Points of Interest (POIs) within it. This is a common
Urban Geography task aimed at understanding the structure of the urban space
(Yao et al., 2017; Zhai et al., 2019; Niu and Silva, 2021; Huang et al., 2022,
2023c).

Figure 2.  The spatial distributions of POI data in the ${UrbanPOI5K}$
dataset.

To quantitively evaluate the performance of LLMs on this urban function
prediction task, we utilize a Points of Interest (POI) dataset from Shenzhen,
China which consists of 303,428 POIs and 5,461 urban neighborhoods with POIs
(Zhang et al., 2017, 2018a; Du et al., 2019, 2020). We denote this dataset as
${UrbanPOI5K}$. Figure 2 shows the geographic distributions of the POIs and
regions. The ground truth data is from the Urbanscape Essential Dataset of
Peking University. The dataset provides detailed spatial distributions of ten
urban function types in the study area: forest, water, unutilized,
transportation, green space, industrial, educational & governmental,
commercial, residential, and agricultural. To simplify the task, we merge the
uncommon urban function types forest, water, unutilized, green space, and
agricultural into the function type outdoors and natural. This results in six
urban function types: (1) residential; (2) commercial; (3) industrial; (4)
education, health care, civic, governmental and cultural, (5) transportation
facilities, and (6) outdoors and natural. In total, 5,344 of the regions have
ground truth data. We randomly split this dataset into training, validation
and test sets with the ratio 60%:20%:20%. The test dataset is used to evaluate
the performance of different models, while the validation set is only used for
supervised baselines.

⬇

[Instruction] There are six land use types: (1) residential, (2) commercial,
(3) industrial, (4) education, health care, civic, governmental and cultural,
(5) transportation facilities, and (6) outdoors and natural.

Paragraph: In this urban region, there are 128 points of interest, including 2
Chinese restaurant, 1 food restaurant, 2 hotel, 2 apartment hotel, 1 daily
life service, 1 mobile communication shop, 24 company, 1 logistics company, 1
real estate agency, 1 lottery retailer, 3 beauty shop, 1 manicure, 2 barber
shop, 4 Internet cafe, 3 bath massage, 2 stadium, 4 training institutions, 1
pharmacy, 4 automative sale, 6 car service, 2 car repair, 1 Car rental, 1
Automobile parts, 3 shopping, 5 shop, 5 parking lot, 5 Parking lot entrance, 2
transportation facility, 1 port harbor, 1 road intersection, 1 atm machine, 2
office building, 2 residential area, 7 building, 1 real estate, 1 park, 1
factory, 7 administrative agency, 1 entrance and exit, 3 gate door, 6
convenience store, 4 home building materials.

Q: What is the primary land use category of this urban region?

A: outdoors and natural

Paragraph:In this urban region, there are 17 points of interest, including 1
food restaurant, 3 public toilet, 3 funeral service, 2 road station for
walking and cycling, 1 beach, 2 parking lot, 2 road intersection, 1 corporate
company enterprise, 2 administrative agency.

Q: What is the primary land use category of this urban region?

A: outdoors and natural

Listing 5: POI-based urban function classification with LLMs, e.g., ChatGPT
(Raw.). Yellow block: the POI statistic of a new urban neighborhood to be
classified. Orange box: ChatGPT (Raw.) outputs.

In order to enable a LLM to handle such a task, we convert the set of POIs
inside an urban region into a textual paragraph that describes the frequencies
of POIs with different place types. Then, we ask the LLM to predict the urban
function of the region based on the paragraph (here we ask for the most
dominating function, in spite of the common presence of mix-used urban
regions). Listing LABEL:ls:prompt-urbanpoi shows one example prompt for this
task, which includes a paragraph-question-answer tuple as a demonstration.
LLMs adapted by this kind of prompt is conducting prediction under a one-shot
setting. For the zero-shot setting, we simply remove this paragraph-question-
answer tuple from the prompt. We use GPT2 with various sizes, GPT-3, and two
ChatGPT models to perform this task under both zero-shot and one-shot
settings. For comparison, we use two supervised learning neural network
baselines:

  * •

Place2Vec: We first learn POI category embeddings following the Place2Vec
method (Yan et al., 2017). Then, given an urban region with $K$ POIs, we
convert each POI into its corresponding Place2Vec embedding and perform mean
pooling to obtain region embeddings as Zhai et al. (Zhai et al., 2019) did.
The resulting neighborhood embeddings are fed into a one-hidden-layer
multilayer perceptron (MLP) to supervise learning its urban function over the
${UrbanPOI5K}$ training dataset.

  * •

HGI: HGI is an unsupervised method for learning region representations based
on POIs. It takes into account the categorical semantics of POIs, as well as
POI-level and region-level adjacency, and the multi-faceted influence from
POIs to regions (Huang et al., 2023c). The HGI region embeddings are fed into
an MLP with the same setup to predict the primary urban function.

Table 5.  Evaluation results of various GPT models and supervised baseline on the ${UrbanPOI5K}$ dataset for the POI-based urban function classification task. We divide the models into three groups: (A) supervised learning-based neural network models; (B) Zero-shot learning with LLMs. (C) One-shot learning with LLMs. We use accuracy, weighted precision, and weighted recall as evaluation metrics. We do not include weighted F1 scores since it is the same as the accuracy score. The best model of each group is highlighted.  | Model | Accuracy | Precision | Recall
---|---|---|---|---
(A) Supervised NN | Place2Vec (Yan et al., 2017; Zhai et al., 2019) | 0.540 | 0.512 | 0.516
HGI (Huang et al., 2023c) | 0.584 | 0.568 | 0.563
(B) Zero-shot LLMs | GPT2 (Radford et al., 2019) | 0.318 | 0.105 | 0.158
GPT2-Medium (Radford et al., 2019) | 0.025 | 0.102 | 0.040
GPT2-Large (Radford et al., 2019) | 0.005 | 0.001 | 0.002
GPT2-XL (Radford et al., 2019) | 0.001 | 0.108 | 0.002
GPT-3 (Brown et al., 2020) | 0.144 | 0.448 | 0.141
ChatGPT (Raw.) (OpenAI, 2022) | 0.075 | 0.376 | 0.106
ChatGPT (Con.) (OpenAI, 2022) | 0.051 | 0.232 | 0.046
(C) One-shot LLMs | GPT2 (Radford et al., 2019) | 0.149 | 0.079 | 0.085
GPT2-Medium (Radford et al., 2019) | 0.317 | 0.104 | 0.156
GPT2-Large (Radford et al., 2019) | 0.057 | 0.083 | 0.021
GPT2-XL (Radford et al., 2019) | 0.324 | 0.105 | 0.159
GPT-3 (Brown et al., 2020) | 0.176 | 0.486 | 0.190
ChatGPT (Raw.) (OpenAI, 2022) | 0.195 | 0.524 | 0.245
ChatGPT (Con.) (OpenAI, 2022) | 0.093 | 0.451 | 0.085

Table 5 shows the evaluation results of all models on the test dataset of
${UrbanPOI5K}$. Additionally, we visualize the confusion matrics of two
baseline models, 7 zero-shot GPT models, and 7 one-shot GPT models in Figure
3, 4, and 5. We can see that:

  * •

In the zero-shot setting, GPT-3 achieves the best precision scores among all
GPT models but still underperforms HGI models.

  * •

Interestingly, in the zero-shot setting, the smallest GPT2 achieves the best
accuracy and recall scores which is counter-intuitive. The reason can be seen
in Figure 4(a). GPT2 predicts almost all neighborhood as “Residential” which
account for 30+% of the ground truth data.

  * •

In the one-shot setting, ChatGPT (Raw.) becomes the best model among all GPT
models in terms of both precision and recall. It achieves 52.4% precision
which is only 4.4% less than HGI . Its confusion matrix in Figure 5(f) also
demonstrates that ChatGPT (Raw.) has reasonably good performance on all urban
function classes.

  * •

In the one-shot setting, GPT2-XL has the highest accuracy. However, Figure
5(d) shows that GPT2-XL is highly biased towards the “Residential” class.

These experimental results highlight the challenges of using LLMs for urban
function classification. Two main reasons contribute to their inadequate
performance:

  * •

POIs are initially used for search in online map services, and by nature, they
contain rich information about commercial venues like restaurants and hotels.
On the contrary, the venues that are not closely related to our daily life,
e.g., factories, are often missing. In this regard, Shenzhen is a heavily
industrial-oriented city, and the ground truth data indicates that there are
many more industrial regions than commercial ones. However, LLMs tend to
predict that a large number of regions are commercial, in view of the
commercial-related POIs fed into it.

  * •

In addition, LLMs are unable to access the spatial distributions of POIs,
which highly influence POI-based urban function prediction since different
spatial distributions of POIs yield different spatial interaction patterns and
thus different urban functions. While both supervised baselines Place2Vec and
HGI are learned from POI distributions during their place type embedding
unsupervised training stage, it is not possible to inform LLMs of the spatial
distributions of POIs. Converting a POI set into an image will also not work.
This is because many POIs will cluster in the downtown area, and a large pixel
size will make a large number of POIs inside one single pixel. On the other
hand, a finer pixel size will make the image of an urban space too large and
cannot be handled by other deep image encoders. Moreover, an urban space image
with a finer pixel size will have very sparse information which is hard for
image encoders to learn. In other words, we need to use specialized neural
architectures to directly handle point data (also polyline data and polygon
data). This calls for the necessity of incorporating encoding architectures of
various geospatial vector data such as location encoding (Mai et al., 2020a,
2022c), polyline encoding (Rao et al., 2020; Yu and Chen, 2022), and polygon
encoding techniques(Mai et al., 2022d) into the GeoAI foundation model
development. We will discuss this in detail in Section 4.6.

(a) Place2Vec

(b) HGI

Figure 3. Confusion matrices of Place2Vec and HGI (Group A in Table 5) on the
${UrbanPOI5K}$ dataset.

(a) GPT2

(b) GPT2-Medium

(c) GPT2-Large

(d) GPT2-XL

(e) GPT-3

(f) ChatGPT (Raw.)

(g) ChatGPT (Con.)

Figure 4. Confusion matrices of all GPT models (Group B in Table 5) on the
${UrbanPOI5K}$ dataset under zero-shot setting.

(a) GPT2

(b) GPT2-Medium

(c) GPT2-Large

(d) GPT2-XL

(e) GPT-3

(f) ChatGPT (Raw.)

(g) ChatGPT (Con.)

Figure 5. Confusion matrices of all GPT models (Group C in Table 5) on the
${UrbanPOI5K}$ dataset under the one-shot setting.

####  3.3.2. Street View Image-Based Urban Noise Intensity Classification

Street view images (SVI) are widely used in many Urban Geography studies to
understand different characteristics of an urban neighborhood such as safety
(Zhang et al., 2018b), beauty, affluence (Lee et al., 2021b), depressing
(Zhang et al., 2018b), housing prices (Kang et al., 2021), noise intensity
levels (Zhao et al., 2023a), accessibility (Hara et al., 2014), etc. It
becomes an important data source that complements remote sensing images.

In this work, we use a recently developed street view image noise intensity
dataset developed by Zhao et al. (Zhao et al., 2023a) as a representative
urban perception task. This dataset consists of 579 street-view images
collected from Singapore. The noise intensity score (between 0 and 1) were
collected based on a human survey. Please refer to their
Github444https://github.com/ualsg/Visual-soundscapes for a detailed
description of this dataset. Since the sound intensity score is not a commonly
agreed metric but an indicator defined by Zhao et al. (Zhao et al., 2023a), it
would be challenging for visual foundation models trained on general web data
such as OpenCLIP (Ilharco et al., 2021) and BLIP (Li et al., 2022) to directly
predict such a score. Therefore, we discretize the original noise intensity
score of each street view image into four classes: very quiet (0 - 0.25),
quiet (0.25 - 0.50), noisy (0.50 - 0.75), and very noisy (0.75 - 1.00). We
denote this dataset as ${SingaporeSVI579}$. Figure 6 illustrates some street
view image examples from each noise intensity class. We randomly split
${SingaporeSVI579}$ into 50% training and 50% testing sets, where the testing
dataset is used to evaluate different CNN and foundation models.

Since all GPT models (except GPT-4) used in previous experiments are pure
language models that cannot handle data modalities such as images. So for the
street view image-based noise intensity prediction task, we select the latest
high-performance open visual-language foundation models (VLFM) including
OpenCLIP (Ilharco et al., 2021), BLIP (Li et al., 2022), and OpenFlamingo-9B
(Awadalla et al., 2023). Although, there exist more powerful visual-language
foundation models such as DeepMind’s Flamingo-9B (Alayrac et al., 2022),
KOSMOS-1 (Huang et al., 2023a), and GPT-4 (OpenAI, 2023), they are not openly
accessible, nor do they provide API access yet555Note that the GPT-4 API still
does not support visual question answering at the time we submit this paper..
We describe the setting of each VLFM as follows:

  * •

OpenCLIP-L: We use an OpenCLIP (Ilharco et al., 2021) ViT L/14 model pre-
trained with the LAION-2B English subset of
LAION-5B666https://laion.ai/blog/laion-5b/ as a small-sized OpenCLIP model. We
download the pre-trained model from
Huggingface777https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K.

  * •

OpenCLIP-B: We use the OpenCLIP (Ilharco et al., 2021) ViT-bigG/14 model
trained with the LAION-2B English subset of LAION-5B as a larger-sized
OpenCLIP model. The pre-trained model is from
Huggingface888https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k.

  * •

BLIP: We use the pre-trained BLIP-2 model (Li et al., 2023) provided by
Huggingface999https://huggingface.co/Salesforce/blip2-flan-t5-xl which
consists of a CLIP-like image encoder, a Querying Transformer (Q-Former), and
a large language model (Flan T5-xl).

  * •

OpenFlamingo-9B: We use the pre-trained OpenFlamingo-9B model (Awadalla et
al., 2023) provided by
Huggingface101010https://huggingface.co/openflamingo/OpenFlamingo-9B which
consists of an image encoder (CLIP ViT-L/14 (Ilharco et al., 2021)) and a
large language model (LLaMA-7B (Touvron et al., 2023)).

All VLFMs are evaluated on the testing set of ${SingaporeSVI579}$ in a zero-
shot setting. Since different VLFMs require different image input formats and
expect different styles of text prompts, we describe the zero-shot pipeline
for each VLFM below:

  * •

OpenCLIP-L and OpenCLIP-B: We first encode four noise intensity class names
into four text embeddings by using a text template of the form “a city area
with the noise intensity of [NOISE_INTENSITY_CLASS]”. Then given a street view
image, we use OpenCLIP ViT image encoder to encode them into an image
embedding. The cosine similarity between this image embedding and all four
class text embeddings are computed and the class with the highest similarity
will be picked as the prediction.

  * •

BLIP: Given a street view image, we use a prompt of the form “What is the
noise intensity of this area, is it 1. very quiet, 2. quiet, 3. noisy, or 4.
very noisy?” to instruct the language encoder of BLIP to predict its noise
intensity class.

  * •

OpenFlamingo-9B: We use a prompt of the form “There are four noise intensity
levels: 1. very quiet, 2. quiet, 3. noisy, or 4. very noisy. <image>The noise
intensity of this area is” to instruct OpenFlamingo-9B to predict the noise
intensity of the given image. Here “<image>” denotes an image token and CLIP
ViT-L/14 is used as the encoder.

We select four convolutional neural network models (CNNs) as the alternative
baselines to compare against these VLFMs: AlexNet (Krizhevsky et al., 2017),
ResNet18 (He et al., 2016b), ResNet50 (He et al., 2016b), and DenseNet161
(Huang et al., 2017). The weights of all CNNs models are first initialized by
the Place365 pre-trained weights (Zhou et al., 2017), and only their final
softmax layers are finetuned with full supervision on the ${SingaporeSVI579}$
training dataset. We choose this linear probing method instead of fully
finetuning the whole CNN architecture due to the very limited training data
size.

Table 6.  Evaluation results of various vision-language foundation models and baselines on the urban street view image-based noise intensity classification dataset, SingaporeSVI579 (Zhao et al., 2023a). We classify models into two groups: (A) Supervised finetuned convolutional neural networks (CNNs); (B) Zero-shot learning with visual-language foundation models (VLFMs). We use accuracy and weighted F1 scores as evaluation metrics. The best scores for each group are highlighted.  | Model | #Param | Accuracy | F1
---|---|---|---|---
(A) Supervised Finetuned CNNs | AlexNet (Krizhevsky et al., 2017) | 58M | 0.452 | 0.405
ResNet18 (He et al., 2016b) | 11M | 0.493 | 0.442
ResNet50 (He et al., 2016b) | 24M | 0.500 | 0.436
DenseNet161 (Huang et al., 2017) | 27M | 0.486 | 0.382
(B) Zero-shot FMs | OpenCLIP-L (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 427M | 0.128 | 0.089
OpenCLIP-B (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 2.5B | 0.169 | 0.178
BLIP (Li et al., 2022, 2023) | 3.9B | 0.452 | 0.405
OpenFlamingo-9B (Awadalla et al., 2023) | 8.3B | 0.262 | 0.127

Table 6 compares the performances of different finetuned CNN models with four
zero-shot VLFMs. The results show that BLIP achieves the best accuracy and
weighted F1 score among all VLFMs in the zero-shot learning setting. The
performance of BLIP is comparable to those of AlexNet but is still slightly
worse than the best model, ResNet18 and ResNet50. To further understand the
classification accuracy of different models on each noise intensity class, we
visualize the confusion matrices of all models in Figure 7. We can see that
the predictions of OpenCLIP-L, OpenCLIP-B, and OpenFlamingo-9B are highly
biased. OpenCLIP-L and OpenCLIP-B tend to classify most street view images as
‘very quiet’ while OpenFlamingo-9B classifies most images as ‘very noisy’. On
the other hand, only BLIP shows balanced and reasonable predictions on all
four noise intensity classes, similar to those fine-tuned CNN models.

(a) Very Quiet (0.100)

(b) Quiet (0.350)

(c) Noisy (0.600)

(d) Very Noisy (0.965)

Figure 6. Some street view image examples in ${SingaporeSVI579}$ dataset. The
image caption indicates the noise intensity class this image belongs to and
the numbers in parenthesis indicate the original noise intensity scores from
Zhao et al. (Zhao et al., 2023a).

(a) AlexNet

(b) ResNet18

(c) ResNet50

(d) DenseNet161

(e) OpenCLIP-L

(f) openclipb

(g) BLIP

(h) OpenFlamingo-9B

Figure 7. Confusion matrices of all baselines and visual-language FMs on
${SingaporeSVI579}$ dataset.

These results are very encouraging with zero-shot BLIP achieving comparable
performance with fine-tuned models. We can observe from Figure 7(g) that BLIP
has a general sense of the noisy intensity level of the target urban area,
e.g., it mis-classifies most “very noisy” areas as simply “noisy”. This
implies that BLIP understands noisy intensity levels on a different scale. For
example, a “very noisy” place annotated by a human interviewee in Singapore
might not qualify as “very” for BLIP, which might have seen many much noisier
urban areas. To this end, BLIPis generally competent for this urban perception
task. At the same time, we recognize that most of the open visual-language
foundation models are still not powerful enough to connect visual features to
their important yet nuanced semantics and concepts in urban studies. For
example, when presented with a construction site in Figure 6(d), we expect a
VLFM to predict that this is a very noisy neighborhood. When seeing a large
vegetation coverage in Figure 6(d), a VLFM should associate this visual
feature with the concept of ‘quiet’ in the language space. This study
highlights the fact that the current VLFMs have certain capabilities in
understanding the characteristics of urban neighborhoods given visual inputs.
However, their ability is still generally not as strong as the current LLMs on
language-only tasks. Furthermore, we think the urban perception task, as a
classic task in urban geography, is more challenging than current visual
question-answering tasks commonly used in VLFM research (Radford et al.,
2021a; Huang et al., 2023a) partly due to their partially subjective nature
and the rarity of annotated datasets. This further emphasizes the unique
challenges faced by foundation model research in GeoAI.

###  3.4. Remote Sensing

Our final experiment focuses on a typical remote sensing (RS) task – remote
sensing image scene classification. We choose a widely-used aerial image scene
classification dataset, ${AID}$ (Xia et al., 2017), which consists of 10K
scenes and 30 aerial scene types. These data were collected from Google Earth
imagery. Please refer to Xia et al. (Xia et al., 2017) for a detailed
description of this dataset. ${AID}$ does not provide an official dataset
split, so we split the dataset into training and testing sets using stratified
sampling with a ratio of 80% for training and 20% for testing, ensuring that
both sets have similar scene type label distributions.

Similar to the street view image classification task from Section 3.3.1, we
use four CNN models (i.e., AlexNet, ResNet18, ResNet50, and DenseNet161) and
four visual-language foundation models (i.e., OpenCLIP-L, OpenCLIP-B, BLIP,
and OpenFlamingo-9B). For all CNNs models, their weights are first initialized
by the ImageNet-V1 pre-trained weights, and their final softmax layers are
fine-tuned with full supervision on the ${AID}$ training dataset. For the
VLFMs, their model performances are highly dependent on whether their language
model component can correctly comprehend the semantics of each RS image scene
type. However, many RS image scene types of ${AID}$ are vague such as
“center”, and “commercial”. We find that if keeping their original scene type
names, models like OpenCLIP would assign no RS image to those two types.
Therefore, we modify the names of “center” to “theater” (although only
partially covers the semantics of this class), and “commercial” to “commercial
area” and use them in the prompt. Models with such prompts are denoted as
“$(Updated)$” while “$(Origin)$” denotes the original RS image scene type
names from ${AID}$ being used in the prompt. We evaluate all VLFMs in a zero-
shot learning setting. Following the street view image classification task in
Section 3.3.1, similar prompt formats are used on the ${AID}$ dataset.

Table 7.  Evaluation results of various vision-language foundation models and baselines on the remote sensing image scene classification dataset, ${AID}$ (Xia et al., 2017). We use the same model set as Table 6. “$(Origin)$” denotes we use the original remote sensing image scene class name from ${AID}$ to populate the prompt while “$(Updated)$”indicates we update some class names to improve its semantic interpretation for FMs. We use accuracy and F1 score as evaluation metrics.  | Model | #Param | Accuracy | F1
---|---|---|---|---
Supervised Finetuned CNNs | AlexNet (Krizhevsky et al., 2017) | 58M | 0.831 | 0.827
ResNet18 (He et al., 2016b) | 11M | 0.752 | 0.730
ResNet50 (He et al., 2016b) | 24M | 0.757 | 0.738
DenseNet161 (Huang et al., 2017) | 27M | 0.818 | 0.807
Zero-shot FMs | OpenCLIP-L $(Origin)$ (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 427M | 0.708 | 0.688
OpenCLIP-L $(Updated)$ (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 427M | 0.710 | 0.698
OpenCLIP-B $(Origin)$ (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 2.5B | 0.699 | 0.668
OpenCLIP-B $(Updated)$ (Radford et al., 2021b; Ilharco et al., 2021; Schuhmann et al., 2022) | 2.5B | 0.705 | 0.686
BLIP $(Origin)$ (Li et al., 2022) | 2.5B | 0.500 | 0.473
BLIP $(Updated)$ (Li et al., 2022) | 2.5B | 0.520 | 0.494
OpenFlamingo-9B (Awadalla et al., 2023) | 8.3B | 0.206 | 0.154

Table 7 summarizes the experiment results of four fine-tuned CNNs models and
zero-shot VLFMs. We can see that AlexNet achieves the best accuracy and F1
score among all CNN models. Surprisingly, OpenCLIP-L $(Updated)$ obtains the
best accuracy and F1 score among all VLFMs. We observe that bigger models do
not necessarily lead to better results in this task. For example, the largest
model, OpenFlamingo-9B only achieves a 0.206 accuracy. One possible reason is
that these larger VLFMs might not see remote-sensing images in their training
data, which usually contain general web-crawled images and texts. OpenCLIP ,
on the other hand, explicitly includes satellite images in their pre-training
data(Ilharco et al., 2021). However, both BLIP and OpenFlamingo-9B did not
mention whether they utilized remote sensing images during the pre-training
stage. Note that street view images are quite similar to Internet images which
are widely used for VLFM pre-training. RS images, on the other hand, such as
satellite images and UAV (unmanned aerial vehicles), are visually
distinguished from Internet photos where the majority of them are captured
using consumers’ digital cameras at the ground level. If the visual encoders
of BLIP and OpenFlamingo-9B are not pre-trained on RS images, the features
they extracted will not align well with text features that share similar
semantics–this leads to poor performance on the ${AID}$ dataset. Our study
highlights the importance of pre-training VLFMs on a diverse set of visual
inputs, including RS images, to improve their performance on remote sensing
tasks.

Another important observation is that the semantics embedded in the prompts
play a pivotal role in determining the model’s performance. For example, when
using the original scene type name “center”, generally none of the models is
able to understand the underlying ambiguous meaning. However, simply changing
“center” to “theater” could help OpenCLIP correctly find relevant RS scenes,
although this is not a perfect name to describe this class. Nevertheless, this
simple change demonstrates the importance of choosing expressive prompts while
using FMs for geospatial tasks.

Compared with the results in Table 5, the experimental results in Table 7
highlight the unique challenges of remote sensing images. We will discuss the
improvement of FMs for remote sensing in detail in Section 4.4.

##  4\. A Multimodal Foundation Model for GeoAI

Section 3 explores the effectiveness of applying existing FMs on different
tasks from various geospatial domains. We can see that many large language
models can outperform fully-supervised task-specific ML/DL models and achieve
surprisingly good performances on several geospatial tasks such as toponym
recognition, location description recognition, and time series forecasting of
dementia. However, on other geospatial tasks (i.e., the two tested Urban
Geography tasks and one RS task), especially those that involve multiple data
modalities (e.g., point data, street view images, RS images, etc.), existing
foundation models still underperform task-specific models. In fact, one unique
characteristic of many geospatial tasks is that they involve many data
modalities such as text data, knowledge graphs, remote sensing images, street
view images, trajectories, and other geospatial vector data. This will put a
significant challenge on GeoAI foundation model development. So in this
section, we discuss the challenges unique to each data modality, then propose
a potential framework for future GeoAI which leverages a multimodal FM.

###  4.1. Geo-Text Data

Despite the promising results showed in Table 1, LLMs still struggle with more
complex geospatial semantics tasks such as toponym resolution/geoparsing (Alex
et al., 2015; Gritta et al., 2018; Wang and Hu, 2019) and geographic question
answering (GeoQA) (Mai et al., 2020b, 2021), since LLMs are unable to perform
(implicit) spatial reasoning in a way that is grounded in the real world. As a
concrete example, we illustrate the shortcomings of GPT-3 on a geoparsing
task. Using two examples from the Ju2016 dataset, we ask GPT-3 to both: 1)
recognize toponyms; and 2) predict their geo-coordinates. The prompt is shown
in List LABEL:ls:prompt-geoparse while the geoparsing results are visualized
in Figure 8. We see that in both cases, GPT-3 can correctly recognize the
toponyms but the predicted coordinates are 500+ miles away from the ground
truth. Moreover, we notice that with a small spatial displacement of the
generated geo-coordinates, GPT-3’s log probability for this new pair of
coordinates decreases significantly. In other words, the probability of
coordinates generated by the LLM does not follow Tobler’s First Law of
Geography (Tobler, 1970). GPT-3 also generates invalid
latitudinal/longitudinal coordinates, indicating that existing LLMs are still
far from gracefully handling complex numerical and spatial reasoning tasks.

Figure 9 provides another example of unsatisfactory results of LLMs in
answering geographic questions related to spatial relations. In this example,
Monore, in the generated answer by the ChatGPT generated answer is not in the
north of Athens, GA, but in the southwest of Athens. This example indicates
that LLMs do not fully understand the semantics of spatial relation. The
reason for this error could be that ChatGPT generates answers to this spatial
relation question based on searching through its internal memory of text-based
knowledge rather than performing spatial reasoning. One potential solution to
this problem could be the use of geospatial knowledge graphs(Zhu et al., 2022;
Cai et al., 2022), which can guide the LLMs to perform explicit spatial
relation computations. We will discuss this further in the next section.

(a) [TEXT]: Franklin is a city in and the county seat of simpson county, …

(b) [TEXT]: the city of Fairview had a population of 260 as of july 1, 2015. …

Figure 8.  Geoparsing examples of GPT-3 on the Ju2016 dataset comparing the
predicted coordinates (dropped pins) and the ground truth coordinates
(starting points). The recognized toponyms are underlined in text.

⬇

[Instruction]...

Paragraph: San Jose was founded in 1803 when allotments of land were made ...

Q: Which words in this paragraph represent named places?

A: San Jose; New Mexico

Q: What is the location of San Jose?

A: 35.39728, -105.47501

...

\--

Paragraph: the city of fairview had a population of 260 as of july 1, 2015.
...

Q: Which words in this paragraph represent named places?

A: Fairview

Q: What is the location of Fairview?

A: 41.85003, -87.65005

Listing 6: Geoparsing with LLMs, e.g., GPT-3. Yellow block: the text snippet
to be geoparsed. Orange box: GPT-3 outputs.  Figure 9. One example in which
ChatGPT gives wrong answers to a geographic question about topological
relations. In this example, Monore is not in the north, but the southwest of
Athens, GA.

###  4.2. Geospatial Knowledge Graph

Despite the superior end-to-end prediction and generation capability, LLMs may
produce content that lacks sufficient coverage of factual knowledge or even
contains non-factual information. To address this problem, knowledge graphs
(KGs) can serve as effective sources of information that complement LLMs. KGs
are factual in nature because the information is usually extracted from
reliable sources, with post-processing conducted by human editors to further
ensure incorrect content is removed. As an important type of domain knowledge
graphs, geospatial knowledge graphs (GeoKG) such as GeoNames (Ahlers, 2013),
LinkedGeoData (Auer et al., 2009), YAGO2 (Hoffart et al., 2013), GNIS-LD
(Regalia et al., 2018), KnowWhereGraph (Janowicz et al., 2022), EVKG (Qi et
al., 2023), etc. are usually generated from authoritative data sources and
spatial databases. For example, GNIS-LD was constructed based on USGS’s
Geographic Names Information System (GNIS). This ensures the authenticity of
these geospatial data.

In particular, developing multimodal FMs for GeoAI which jointly consider text
data and geospatial knowledge graphs can lead to several advantages. First,
from the model perspective, (geospatial) knowledge graphs could be integrated
into pre-training or fine-tuning LLMs, through strategies such as retrieving
embeddings of knowledge entities for contextual representation learning
(Peters et al., 2019), fusing knowledge entities and text information (He et
al., 2021b; Zhang et al., 2022a), designing learning objectives that focus on
reconstructing knowledge entities (Zhang et al., 2019) and triples (Yasunaga
et al., 2022). Second, from the data perspective, GeoKGs could provide
contextualized semantic and spatiotemporal knowledge to facilitate prompt
engineering or data generation, such as enriching prompts with contextual
information from KGs (Brate et al., 2022; Wu et al., 2023) and converting KG
triples into natural text corpora for specific domains (Agarwal et al., 2021).
Third, from the application perspective, it is possible to convert facts in
geospatial knowledge graphs into natural language to enhance text generation
(Yu et al., 2022), to be used in scenarios such as (geographic) question
answering (Fan et al., 2019; Mai et al., 2019) and dialogue systems (Wu et
al., 2019b). Last, from a reasoning perspective, GeoKGs usually provide
spatial footprints of geographic entities which enable LLMs to perform
explicit spatial reasoning as Neural Symbolic Machine did (Liang et al.,
2017). This can help avoid the errors we see in Figure 9.

###  4.3. Street View Image

Section 3.3.1 has demonstrated the effectiveness of existing visual-language
foundation models on a street view-based geospatial task. However, the
performance gaps between the task-specific models and VLFMs shown in Table 6
inform us that there are some unique characteristics of urban perception tasks
we need to consider if we want to develop a FM for GeoAI.

Although street view images are like the natural images used in common vision-
language tasks, one major difference is that common vision-language tasks
usually focus on factual knowledge in images (e.g., “how many cars in this
image”) while urban perception tasks are usually related to high-level human
perception of the images such as the safety, poverty, beauty, and sound
intensity of a neighborhood given a street view image. Compared with factual
knowledge, this kind of high-level perception knowledge is rather hard to
estimate and the labels are rather rare. Moreover, many perception concepts
are vague and subjective which increases the difficulties of those tasks. So
in order to develop a GeoAI FM that can achieve state-of-the-art performances
on various urban perception tasks, we need to conduct some domain studies to
provide a concrete definition of each urban perception concept and develop
some annotated datasets for GeoAI FM pre-training.

###  4.4. Remote Sensing

With the advancement of computer vision technology, deep vision models have
been successfully applied to different kinds of remote sensing (RS) tasks
including image classification/regression (Ayush et al., 2021; Rolf et al.,
2021), land cover classification (Ayush et al., 2021), and object
detection(Lam et al., 2018). Unlike the usual vision tasks which usually work
on RGB images, RS tasks are based on multispectral/hyperspectral images from
different sensors. Most existing RS works focus on training one model for a
specific RS task using data from a specific sensor (Lam et al., 2018).
Researchers often compare performances of different models using the same
training datasets and decide on model implementation based on accuracy
statistics. However, we see the trend of FMs in the CV field such as CLIP
(Radford et al., 2021a), Flamingo-9B (Alayrac et al., 2022) to be further
developed to meet the unique challenges of remote sensing tasks. RS
experiments in Section 3.4 demonstrate that there is still a performance gap
between current visual-language foundation models and task-specific deep
models. To fill this gap and develop a GeoAI FM that can achieve state-of-the-
art performances on various RS tasks, we need to consider the uniqueness of RS
images and tasks.

Aside from being task-agnostic, the desiderata for a remote sensing FM include
being: 1) sensor-agnostic: it can seamlessly reason among RS images from
different sensors with different spatial or spectral resolutions; 2)
spatiotemporally-aware: it can handle the spatiotemporal metadata of RS images
and perform geospatial reasoning for tasks such as image geolocalization and
object tracking; 3) environmentally-invariant: it can decompose and isolate
the spectral characteristics of the objects of interest across a variety of
background environmental conditions and landscape structure. Recent
developments here include geography-aware RS models (Ayush et al., 2021) or
self-supervised/unsupervised RS models (Ayush et al., 2021; Rolf et al.,
2021), all of which are task-agnostic. However, we have yet to develop a FM
for RS tasks which can satisfy all such properties.

In summary, efforts should be focused on developing GeoAI FMs using remote
sensing to address pressing environmental challenges due to climate change. It
would require complex models which look beyond image classification toward
modeling ecosystem functions such as forest structure, carbon sequestration,
urban heat, coastal flooding, and wetland health. Traditionally remote sensing
is widely used to study these phenomena but in a site-specific and sensor-
specific manner. Sensor-agnostic, spatiotemporally-aware, and environmentally-
invariant FMs have the potential to transform our understanding of the trends
and behavior of these complex environmental phenomena.

###  4.5. Trajectory and Human Mobility

Trajectory, which is a sequence of time-ordered location tuples, is another
important data type in GeoAI. The proliferation of digital trajectory data
generated from various sensors (e.g., smartphones, wearable devices, and
vehicle on-board devices) together with the advancement of deep learning
approaches has enabled novel GeoAI models for modeling human mobility
patterns, which are crucial for city management and transportation services,
etc. There are four typical tasks in modeling human dynamics with deep
learning (Luca et al., 2021), including trajectory generation (Rao et al.,
2020; Choi et al., 2021), origin-destination (OD) flow generation (Yao et al.,
2020; Simini et al., 2021), in/out population flow prediction (Li et al.,
2021; Jiang et al., 2021), and next-location/place prediction (Rao et al.,
2021; Lin et al., 2021).

In order to develop GeoAI FMs for supporting human mobility analysis, we need
to consider the following perspectives: 1) pre-training and generation of
task-agnostic trajectory embedding (Wang et al., 2021; Musleh et al., 2022),
which represent high-level movement semantics (e.g., spatiotempporal
awareness, routes, and location sequence) from various kinds of trajectories
(Lin et al., 2021); 2) context-aware contrastive learning of trajectory: human
movements are constrained from their job type, surrounding built environment,
and transportation infrastructure as well as many other spatiotemporal and
environmental factors (Siła-Nowicka et al., 2016; Wang et al., 2018; Luca et
al., 2021); GeoAI FMs should be able to link trajectories to various
contextual representations such as road networks (e.g., Road2Vec (Liu et al.,
2017), (Chen et al., 2021a)), POI composition or land use types (Zhang et al.,
2021), urban morphology (Chen et al., 2021b), and population distribution
(Huang et al., 2021); 3) user geoprivacy (Keßler and McKenzie, 2018) should be
protected when training such GeoAI FMs since trajectory data can reveal
individuals’ sensitive locations such as home and personal trips. The privacy-
preserving techniques by utilizing cryptography or differential privacy (Al-
Rubaie and Chang, 2019) and federated learning framework may be incorporated
in the GeoAI FMs training process for trajectories (Rao et al., 2021).

###  4.6. Geospatial Vector Data

Another critical challenge in developing FMs for GeoAI is the complexity of
geospatial vector data which are commonly used in almost all GIS and mapping
platforms. Examples include the US state-level and county-level dementia data
(polygon data) discussed in Section 3.2, urban POI data (point and polygon
data) introduced in Section 3.3.1, cartographic polyline data (Yu and Chen,
2022), building footprints data (Mai et al., 2022d), road networks (composed
by points and polylines), and many others. In contrast with NLP and CV where
text (1-D) or images (2-D) are well-structured and more suitable to common
neural network architectures, vector data exhibits more complex data
structures in the form of points, polylines, polygons, and networks (Mai et
al., 2022c). So it is particularly challenging to develop a FM which can
seamlessly encode or decode different kinds of vector data.

Noticeably, recently developed location encoding (Mai et al., 2020a, 2022c),
polyline encoding (Rao et al., 2020; Yu and Chen, 2022), and polygon encoding
techniques(Mai et al., 2022d) can be seen as a fundamental building block for
such a model. Moreover, since encoding (e.g., geo-aware image
classification(Mai et al., 2020a)) or decoding (e.g., geoparsing (Wang and Hu,
2019)) geospatial vector data, or conducting spatial reasoning (e.g., GeoQA
(Mai et al., 2021)) is an indispensable component for most GeoAI tasks,
developing FMs for vector data is the key step towards a multimodal FM for
GeoAI. This point also differentiates GeoAI FMs from existing FMs in other
domains.

###  4.7. A Multimodal FM for GeoAI

Except for those data modalities, there are also other datasets frequently
studied in GeoAI such as geo-tagged videos, spatial social networks, sensor
networks, and so on. Given all these diverse data modalities, the question is
how to develop a multimodal FM for GeoAI that best integrates all of them.

When we take a look at the existing multimodal FMs such as CLIP (Radford et
al., 2021a), DALL$\cdot$E2 (Ramesh et al., 2022), MDETR (Kamath et al., 2021),
VATT (Akbari et al., 2021), BLIP (Li et al., 2022), DeepMind Flamingo (Alayrac
et al., 2022), KOSMOS-1 (Huang et al., 2023a), we can see the following
general architecture: 1) starting with separate embedding modules to encode
different modalities of data (e.g., a Transformer for texts and ViT for images
(Radford et al., 2021a)); 2) (optionally) mixing the representations of
different modalities by concatenation; 3) (optionally) more Transformer layers
for across modality reasoning, which can achieve a certain degree of alignment
based on semantics, e.g., the word “hospital” attending to a picture of a
hospital; 4) generative or discriminative prediction modules for different
modalities to achieve self-supervised training.

One weak point of these architectures is the lack of integration with
geospatial vector data, which is the backbone of spatial reasoning and helps
alignment among multi-modalities in GeoAI. This is considered central and
critical for GeoAI tasks. Therefore, we propose to replace step 2 with
aligning the representations of different modalities (e.g., geo-tagged texts
and RS images) by augmenting their representations with location encoding(Mai
et al., 2020a) before mixing them. Figure 10 illustrates this idea. Geo-tagged
text data, street view images, remote sensing images, trajectories, and
geospatial knowledge graphs can be easily aligned via their geographic
footprints (vector data). The key advantages of such a model are to enable
spatial reasoning and knowledge transfer across modalities.

Figure 10.  A multimodal FM which achieves alignment among different data
sources via their geospatial relationships.

##  5\. Risks and Challenges

Despite the recent progress, several challenges are emerging as more advanced
FMs have been released (Zhao et al., 2023b). First, as FMs continue to
increase in size, there is a need to improve the computational efficiency for
training and fine-tuning these models. Second, as an increasing number of LLMs
are not open-sourced, it becomes challenging to incorporate knowledge into
these models without accessing to their internal parameters. Third, as LLMs
are increasingly deployed in remote third-party settings, protecting user
privacy becomes increasingly important.

Except for these challenges for FMs in general, there are also many unique
challenges and risks during the process of GeoAI FMs development.

###  5.1. Geographic Fidelity

Many FMs are criticized for generating inaccurate and misleading results
(Touvron et al., 2023; Huang et al., 2023b). In a geographic context,
generating geographic faithful results is particularly important for almost
all GeoAI tasks. In addition to Figure 9 in Section 4.1, Figure 11 illustrates
two geographically inaccurate results generated from ChatGPT and Stable
Diffusion. In Figure 11(a), the expected answer should be “Washington, North
Carolina”111111https://en.wikipedia.org/wiki/Washington,_North_Carolina.
However, ChatGPT indicates there is no Washington in North Carolina. Moreover,
the largest city in Washington State should be Seattle and there is no city in
this state named Washington. Figure 11(b) visualizes 4 generated remote
sensing images generated by Stable
Diffusion121212https://huggingface.co/spaces/stabilityai/stable-diffusion.
Although those images appear similar to satellite images, it is rather easy to
tell that they are fake RS images since the layouts of geographic features in
these images are clearly not from any city in the world. In fact, generating
faithful RS images is a popular and important RS task (Han et al., 2021; He et
al., 2021a) in which geometric accuracy is very important for the downstream
tasks.

(a) Geographically inaccurate results from ChatGPT

(b) Geographically inaccurate results from Stable Diffusion

Figure 11. Some geographically inaccurate results generated from different
language and vision foundation models. (a) The expected answer “Washington,
North Carolina” is not generated correctly. Moreover, there is no city in the
state of Washington. The largest city in Washington State should be Seattle.
(b) The generated remote sensing images from Stable Diffusion do not have
correct geographic layouts such as road networks, waterbodies, etc.

###  5.2. Geographic Bias

It is well known that foundation models have the potential to amplify existing
societal inequalities and biases present in the data (Bommasani et al., 2021;
Zhang et al., 2022b; Touvron et al., 2023). A key consideration for GeoAI in
particular is _geographic bias_ (Liu et al., 2022a), which is often overlooked
by AI research. For example, Zilong et al. (Liu et al., 2022a) showed that all
current geoparsers are highly geographically biased towards data-rich regions.
The same issue can be observed in current LLMs. Figure 12 shows two examples
in which both ChatGPT and GPT-4 generate inaccurate results due to the
geographic bias inherited in these models. Compared with San Jose, California,
USA, San Jose, Batangas131313https://en.wikipedia.org/wiki/San_Jose,_Batangas
is a less popular place name in many text corpus. Similarly, compared with
Washington State, USA and Washington, D.C., USA, Washington, New
York141414https://en.wikipedia.org/wiki/Washington,_New_York is also a less
popular place name. That is why both ChatGPT and GPT-4 interpret those place
names incorrectly. Compared to task-specific models, FMs suffer more from
geographic bias since: 1) the training data is collected in large-scale which
is likely to be dominated by overrepresented communities or regions; 2) the
huge number of learnable parameters and complex model structures make model
interpretation and debiasing much more difficult; 3) the geographic bias of
the FMs can be easily inherited by all the adapted models downstream
(Bommasani et al., 2021), and thus bring much more harm to the society. This
indicates a pressing need for designing proper (geographic) debiasing
frameworks.

(a) Inaccurate results generated by ChatGPT due to geographic bias

(b) Inaccurate results generated by GPT-4 due to geographic bias

Figure 12. Some inaccurate results generated from different ChatGPT and GPT-4
due to geographic bias. (a) San Jose, California, USA is a more popular place
name compared with San Jose, Batangas. So ChatGPT interprets the name “San
Jose” incorrectly and leads to a wrong answer. (b) Washington State, USA and
Washington, D.C., USA are two popular places with name “Washington”. The
correct answer “Washington, New York” is less popular which leads to an
inaccurate answer.

###  5.3. Temporal Bias

Similar to geographic bias, FMs also suffer from temporal bias since there is
much more training data available for current geographic entities than for
historical ones Temporal bias can also lead to inaccurate results. Two
examples are shown in Figure 13. In both cases, the names of historical places
are used for other places nearby. GPT-4 fails to answer both questions due to
its heavy reliance on pre-training data which are biased towards current
geographic knowledge. Temporal bias and geographic bias are critical
challenges that need to be solved for the development of GeoAI FMs.

(a) Inaccurate results generated by ChatGPT due to temporal bias

(b) Inaccurate results generated by GPT-4 due to temporal bias

Figure 13. Some inaccurate results generated from GPT-4 due to temporal bias.
(a) Flagler Beach, Florida used to be named as Ocean City during 1913 – 1923
while Ocean City, Florida now is used to call another place in Florida. GPT-4
fails to recognize this and return a wrong answer. (b) Fountain City, Indiana
was named by Newport during 1834 - 1878 while now Newport is used to call
another city, Newport, Indiana in Vermillion County. GPT-4 fails to answer it
correctly.

###  5.4. Spatial Scale

Geographic information can be represented in different spatial scales, which
means that the same geographic phenomenon/object can have completely different
spatial representations (points vs. polygons) across GeoAI tasks. For example,
an urban traffic forecasting model must represent San Francisco (SF) as a
complex polygon, while a geoparser usually represents SF as a single point.
Since FMs are developed for a diverse set of downstream tasks, they need to be
able to handle geospatial information with different spatial scales, and infer
the right spatial scale to use given a downstream task. Developing such a
module is a critical component for an effective GeoAI FM.

###  5.5. Generalizability v.s. Spatial Heterogeneity

An open problem for GeoAI is how to achieve model generalizability
(“replicability” (Goodchild and Li, 2021)) across space while still allowing
the model to capture spatial heterogeneity. Given geospatial data with
different spatial scales, we desire a FM that can learn general spatial trends
while still memorizing location-specific details. Will this generalizability
introduce unavoidable intrinsic model bias in downstream GeoAI tasks? Will
this memorized localized information lead to an overly complicated prediction
surface for a global prediction problem? With large-scale training data, this
problem can be amplified and requires care .

##  6\. Conclusion

In this paper, we explore the promises and challenges for developing
multimodal foundation models (FMs) for GeoAI. The potential of FMs is
demonstrated by comparing the performance of existing LLMs and visual-language
FMs as zero-shot or few-shot learners with fully-supervised task-specific SOTA
models on seven tasks across multiple geospatial subdomains such as Geospatial
Semantics, Health Geography, Urban Geography, and Remote Sensing. While in
some language-only geospatial tasks, LLMs, as zero-shot or few-shot learners,
can outperform task-specific fully-supervised models, existing FMs still
underperform the task-specific fully-supervised models on other geospatial
tasks, especially tasks involving multiple data modalities (e.g., POI-based
urban function classification, street view image-based urban noise intensity
classification, and remote sensing image scene classification). We realize
that the major challenge for developing a FM for GeoAI is the multimodality
nature of geospatial tasks. After discussing the unique challenges of each
geospatial data modality, we propose our vision for a novel multimodal FM for
GeoAI that should be pre-trained based on the alignment among different data
modalities via their geospatial relations. We conclude this work by discussing
some unique challenges and risks for such a model.

## References

  * (1)
  * Agarwal et al. (2021) Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021\.  Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 3554–3565.
  * Ahlers (2013) Dirk Ahlers. 2013\.  Assessment of the accuracy of GeoNames gazetteer data. In _Proceedings of the 7th workshop on geographic information retrieval_. 74–81.
  * Akbari et al. (2021) Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. 2021.  Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text.  _Advances in Neural Information Processing Systems_ 34 (2021), 24206–24221.
  * Akushevich et al. (2021) Igor Akushevich, Arseniy P Yashkin, Anatoliy I Yashin, and Julia Kravchenko. 2021.  Geographic disparities in mortality from Alzheimer’s disease and related dementias.  _Journal of the American Geriatrics Society_ 69, 8 (2021), 2306–2315.
  * Al-Rubaie and Chang (2019) Mohammad Al-Rubaie and J Morris Chang. 2019.  Privacy-preserving machine learning: Threats and solutions.  _IEEE Security & Privacy_ 17, 2 (2019), 49–58.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022\.  Flamingo: a Visual Language Model for Few-Shot Learning.  _ArXiv_ abs/2204.14198 (2022).
  * Alex et al. (2015) Beatrice Alex et al. 2015\.  Adapting the Edinburgh geoparser for historical georeferencing.  _International Journal of Humanities and Arts Computing_ 9, 1 (2015).
  * Association et al. (2021) Alzheimer’s Association et al. 2021\.  Changing the trajectory of Alzheimer’s disease: how a treatment by 2025 saves lives and dollars. 2015.  _URL: https://www. alz. org/media/Documents/changing-the-trajectory-r. pdf [accessed 2018-07-18][WebCite Cache ID 710WNv2LM]_ (2021).
  * Association et al. (2022) Alzheimer’s Association et al. 2022\.  Alzheimer’s disease facts and figures. More Than Normal Aging: Understanding Mild Cognitive Impairment. Alzheimer’s Association.
  * Auer et al. (2009) Sören Auer, Jens Lehmann, and Sebastian Hellmann. 2009\.  Linkedgeodata: Adding a spatial dimension to the web of data. In _The Semantic Web-ISWC 2009: 8th International Semantic Web Conference, ISWC 2009, Chantilly, VA, USA, October 25-29, 2009. Proceedings 8_. Springer, 731–746.
  * Awadalla et al. (2023) Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023.  _OpenFlamingo_.  https://doi.org/10.5281/zenodo.7733589
  * Ayush et al. (2021) Kumar Ayush et al. 2021\.  Geography-aware self-supervised learning. In _CVPR 2021_. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 10181–10190.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Brate et al. (2022) Ryan Brate, Minh-Hoang Dang, Fabian Hoppe, Yuan He, Albert Meroño-Peñuela, and Vijay Sadashivaiah. 2022.  Improving Language Model Predictions via Prompts Enriched with Knowledge Graphs. In _Workshop on Deep Learning for Knowledge Graphs (DL4KG@ ISWC2022)_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Burke et al. (2021) Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon. 2021.  Using satellite imagery to understand and promote sustainable development.  _Science_ 371, 6535 (2021), eabe8628.
  * Cai et al. (2020) Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, and Rui Zhu. 2020.  Traffic transformer: Capturing the continuity and periodicity of time series for traffic forecasting.  _Transactions in GIS_ 24, 3 (2020), 736–755.
  * Cai et al. (2022) Ling Cai, Krzysztof Janowicz, Rui Zhu, Gengchen Mai, Bo Yan, and Zhangyu Wang. 2022\.  HyperQuaternionE: A hyperbolic embedding model for qualitative spatial and temporal reasoning.  _GeoInformatica_ (2022), 1–39.
  * Chang et al. (2021) Serina Chang, Emma Pierson, Pang Wei Koh, Jaline Gerardin, Beth Redbird, David Grusky, and Jure Leskovec. 2021.  Mobility network models of COVID-19 explain inequities and inform reopening.  _Nature_ 589, 7840 (2021), 82–87.
  * Chen et al. (2021b) Wangyang Chen, Abraham Noah Wu, and Filip Biljecki. 2021b.  Classification of urban morphology with deep learning: Application on urban vitality.  _Computers, Environment and Urban Systems_ 90 (2021), 101706.
  * Chen et al. (2021a) Yile Chen, Xiucheng Li, Gao Cong, Zhifeng Bao, Cheng Long, Yiding Liu, Arun Kumar Chandran, and Richard Ellison. 2021a.  Robust road network representation learning: When traffic patterns meet traveling semantics. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 211–220.
  * Choi et al. (2021) Seongjin Choi, Jiwon Kim, and Hwasoo Yeo. 2021.  Trajgail: Generating urban vehicle trajectories using generative adversarial imitation learning.  _Transportation Research Part C: Emerging Technologies_ 128 (2021), 103091\.
  * DeLozier et al. (2016) Grant DeLozier, Benjamin Wing, Jason Baldridge, and Scott Nesbit. 2016.  Creating a novel geolocation corpus from historical texts. In _LAW-X 2016_. 188–198.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 248–255.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  _ICLR_ (2021).
  * Du et al. (2019) Shouji Du, Shihong Du, Bo Liu, and Xiuyuan Zhang. 2019\.  Context-enabled extraction of large-scale urban functional zones from very-high-resolution images: A multiscale segmentation approach.  _Remote Sensing_ 11, 16 (2019), 1902.
  * Du et al. (2020) Shouji Du, Shihong Du, Bo Liu, Xiuyuan Zhang, and Zhijia Zheng. 2020.  Large-scale urban functional zone mapping by integrating remote sensing images and open social data.  _GIScience & Remote Sensing_ 57, 3 (2020), 411–430.
  * Elmustafa et al. (2022) Amna Elmustafa, Erik Rozi, Yutong He, Gengchen Mai, Stefano Ermon, Marshall Burke, and David Lobell. 2022.  Understanding economic development in rural Africa using satellite imagery, building footprints and deep models. In _Proceedings of the 30th International Conference on Advances in Geographic Information Systems_. ACM, 1–4.
  * Fan et al. (2019) Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019.  Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs. In _2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing_.
  * Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher D Manning. 2005\.  Incorporating non-local information into information extraction systems by gibbs sampling. In _Proceedings of the 43rd annual meeting of the association for computational linguistics (ACL’05)_. 363–370.
  * Goodchild and Li (2021) Michael F Goodchild and Wenwen Li. 2021.  Replication across space and time must be weak in the social and environmental sciences.  _PNAS_ 118, 35 (2021).
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020\.  Generative adversarial networks.  _Commun. ACM_ 63, 11 (2020), 139–144.
  * Gritta et al. (2018) Milan Gritta, Mohammad Taher Pilehvar, and Nigel Collier. 2018.  Which Melbourne? Augmenting Geocoding with Maps. In _ACL 2018_. 1285–1296.
  * Han et al. (2021) Xiaolin Han, Huan Zhang, Jing-Hao Xue, and Weidong Sun. 2021\.  A spectral–spatial jointed spectral super-resolution and its application to HJ-1A satellite images.  _IEEE Geoscience and Remote Sensing Letters_ 19 (2021), 1–5.
  * Hara et al. (2014) Kotaro Hara, Jin Sun, Robert Moore, David Jacobs, and Jon Froehlich. 2014.  Tohme: Detecting Curb Ramps in Google Street View Using Crowdsourcing, Computer Vision, and Machine Learning. In _Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology_ (Honolulu, Hawaii, USA) _(UIST ’14)_. Association for Computing Machinery, New York, NY, USA, 189–204.  https://doi.org/10.1145/2642918.2647403
  * He et al. (2016a) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016a.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2016b) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016b.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * He et al. (2021b) Lei He, Suncong Zheng, Tao Yang, and Feng Zhang. 2021b.  Klmo: Knowledge graph enhanced pretrained language model with fine-grained relationships. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. 4536–4542.
  * He et al. (2021a) Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David Lobell, and Stefano Ermon. 2021a.  Spatial-temporal super-resolution of satellite imagery via conditional pixel synthesis.  _Advances in Neural Information Processing Systems_ 34 (2021), 27903–27915.
  * Hernandez et al. (2021) Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021.  Scaling Laws for Transfer.  arXiv:2102.01293 [cs.LG]
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Hoffart et al. (2013) Johannes Hoffart, Fabian M Suchanek, Klaus Berberich, and Gerhard Weikum. 2013.  YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia.  _Artificial intelligence_ 194 (2013), 28–61.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022\.  Training Compute-Optimal Large Language Models.  _CoRR_ abs/2203.15556 (2022).
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  _To appear_ 7, 1 (2017), 411–420.
  * Hu et al. (2020) Yingjie Hu, , et al. 2020\.  How Do People Describe Locations During a Natural Disaster: An Analysis of Tweets from Hurricane Harvey. In _GIScience 2020_.
  * Hu (2018) Yingjie Hu. 2018\.  Geo-text data and data-driven geospatial semantics.  _Geography Compass_ 12, 11 (2018), e12404.
  * Hu et al. (2014) Yingjie Hu et al. 2014\.  Improving wikipedia-based place name disambiguation in short texts using structured data from dbpedia. In _GIR Workshop 2014_. 1–8.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017.  Densely connected convolutional networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4700–4708.
  * Huang et al. (2023a) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023a.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_ (2023).
  * Huang et al. (2023b) Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023b.  Language is not all you need: Aligning perception with language models.  _arXiv preprint arXiv:2302.14045_ (2023).
  * Huang et al. (2022) Weiming Huang, Lizhen Cui, Meng Chen, Daokun Zhang, and Yao Yao. 2022.  Estimating urban functional distributions with semantics preserved POI embedding.  _International Journal of Geographical Information Science_ 36, 10 (2022), 1905–1930.
  * Huang et al. (2023c) Weiming Huang, Daokun Zhang, Gengchen Mai, Xu Guo, and Lizhen Cui. 2023c.  Learning urban region representations with POIs and hierarchical graph infomax.  _ISPRS Journal of Photogrammetry and Remote Sensing_ 196 (2023), 134–145.
  * Huang et al. (2021) Xiao Huang, Di Zhu, Fan Zhang, Tao Liu, Xiao Li, and Lei Zou. 2021.  Sensing population distribution from satellite imagery via deep learning: Model selection, neighboring effects, and systematic biases.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 14 (2021), 5137–5151.
  * Ilharco et al. (2021) Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021.  _OpenCLIP_.  https://doi.org/10.5281/zenodo.5143773 If you use this software, please cite it as below..
  * Janowicz et al. (2020) Krzysztof Janowicz, Song Gao, Grant McKenzie, Yingjie Hu, and Budhendra Bhaduri. 2020.  GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond.  _International Journal of Geographical Information Science_ 34, 4 (2020), 625–636.
  * Janowicz et al. (2022) Krzysztof Janowicz, Pascal Hitzler, Wenwen Li, Dean Rehberger, Mark Schildhauer, Rui Zhu, Cogan Shimizu, Colby K Fisher, Ling Cai, Gengchen Mai, et al. 2022\.  Know, Know Where, KnowWhereGraph: A densely connected, cross-domain knowledge graph and geo-enrichment service stack for applications in environmental intelligence.  _AI Magazine_ 43, 1 (2022), 30–39.
  * Janowicz et al. (2012) Krzysztof Janowicz, Simon Scheider, Todd Pehle, and Glen Hart. 2012\.  Geospatial semantics and linked spatiotemporal data–Past, present, and future.  _Semantic Web_ 3, 4 (2012), 321–332.
  * Jenkins et al. (2011) Gwilym M Jenkins, George EP Box, and Gregory C Reinsel. 2011\.  _Time series analysis: forecasting and control_. Vol. 734.  John Wiley & Sons.
  * Jiang et al. (2021) Renhe Jiang, Zekun Cai, Zhaonan Wang, Chuang Yang, Zipei Fan, Quanjun Chen, Kota Tsubouchi, Xuan Song, and Ryosuke Shibasaki. 2021.  Deepcrowd: A deep model for large-scale citywide crowd density and flow prediction.  _IEEE Transactions on Knowledge and Data Engineering_ 35, 1 (2021), 276–290.
  * Jones and Purves (2008) Christopher B Jones and Ross S Purves. 2008.  Geographical information retrieval.  _International Journal of Geographical Information Science_ 22, 3 (2008), 219–228.
  * Ju et al. (2016) Yiting Ju et al. 2016\.  Things and strings: improving place name disambiguation from short texts by combining entity co-occurrence with topic modeling. In _EKAW 2016_. Springer, 353–367.
  * Jurafsky and Martin (2009) Dan Jurafsky and James H. Martin. 2009.  _Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd Edition_.  Prentice Hall, Pearson Education International.
  * Kamath et al. (2021) Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021\.  Mdetr-modulated detection for end-to-end multi-modal understanding. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1780–1790.
  * Kang et al. (2023) Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. 2023.  Scaling up gans for text-to-image synthesis.  _arXiv preprint arXiv:2303.05511_ (2023).
  * Kang et al. (2021) Yuhao Kang, Fan Zhang, Wenzhe Peng, Song Gao, Jinmeng Rao, Fabio Duarte, and Carlo Ratti. 2021.  Understanding house price appreciation using multi-source big geo-data and machine learning.  _Land Use Policy_ 111 (2021), 104919.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling Laws for Neural Language Models.  _CoRR_ abs/2001.08361 (2020).
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 4401–4410.
  * Kearns and Moon (2002) Robin Kearns and Graham Moon. 2002.  From medical to health geography: novelty, place and theory after a decade of change.  _Progress in human geography_ 26, 5 (2002), 605–625.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT 2019_. 4171–4186.
  * Keßler and McKenzie (2018) Carsten Keßler and Grant McKenzie. 2018.  A geoprivacy manifesto.  _Transactions in GIS_ 22, 1 (2018), 3–19.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014.  Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.  arXiv:http://arxiv.org/abs/1312.6114v10 [stat.ML]
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023\.  Segment Anything.  _arXiv preprint arXiv:2304.02643_ (2023).
  * Kramarow and Tejada-Vera (2019) Ellen A Kramarow and Betzaida Tejada-Vera. 2019.  Dementia mortality in the United States, 2000-2017.  _National Vital Statistics Reports: From the Centers for Disease Control and Prevention, National Center for Health Statistics, National Vital Statistics System_ 68, 2 (2019), 1–29.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017\.  Imagenet classification with deep convolutional neural networks.  _Commun. ACM_ 60, 6 (2017), 84–90.
  * Kuhn (2005) Werner Kuhn. 2005\.  Geospatial semantics: why, of what, and how?. In _Journal on data semantics III_. Springer, 1–24.
  * Lam et al. (2018) Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. 2018\.  xview: Objects in context in overhead imagery.  _arXiv preprint arXiv:1802.07856_ (2018).
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016\.  Neural architectures for named entity recognition.  _arXiv preprint arXiv:1603.01360_ (2016).
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017\.  Photo-realistic single image super-resolution using a generative adversarial network. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4681–4690.
  * Lee et al. (2021a) Jihyeon Lee, Nina R Brooks, Fahim Tajwar, Marshall Burke, Stefano Ermon, David B Lobell, Debashish Biswas, and Stephen P Luby. 2021a.  Scalable deep learning to identify brick kilns and aid regulatory capacity.  _Proceedings of the National Academy of Sciences_ 118, 17 (2021), e2018863118.
  * Lee et al. (2021b) Jihyeon Lee, Dylan Grosz, Sicheng Zeng, Burak Uzkent, Marshall Burke, David Lobell, and Stefano Ermon. 2021b.  Predicting livelihood indicators from crowdsourced street level images. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Li et al. (2023) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023\.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022\.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_. PMLR, 12888–12900.
  * Li et al. (2021) Mingxiao Li, Song Gao, Feng Lu, Kang Liu, Hengcai Zhang, and Wei Tu. 2021\.  Prediction of human activity intensity using the interactions in physical and social spaces through graph convolutional networks.  _International Journal of Geographical Information Science_ 35, 12 (2021), 2489–2516.
  * Liang et al. (2017) Chen Liang, Jonathan Berant, Quoc Le, Kenneth Forbus, and Ni Lao. 2017.  Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. 23–33.
  * Lin et al. (2021) Yan Lin, Huaiyu Wan, Shengnan Guo, and Youfang Lin. 2021\.  Pre-training context and time aware location embeddings from spatial-temporal trajectories for user next location prediction. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4241–4248.
  * Liu et al. (2017) Kang Liu, Song Gao, Peiyuan Qiu, Xiliang Liu, Bo Yan, and Feng Lu. 2017\.  Road2vec: Measuring traffic interactions in urban road system from massive travel routes.  _ISPRS International Journal of Geo-Information_ 6, 11 (2017), 321.
  * Liu et al. (2022a) Zilong Liu, Krzysztof Janowicz, Ling Cai, Rui Zhu, Gengchen Mai, and Meilin Shi. 2022a.  Geoparsing: Solved or Biased? An Evaluation of Geographic Biases in Geoparsing.  _AGILE: GIScience Series_ 3 (2022), 9.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _Proceedings of the IEEE/CVF international conference on computer vision_. 10012–10022.
  * Liu et al. (2022b) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. 2022b.  A ConvNet for the 2020s.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2022).
  * Luca et al. (2021) Massimiliano Luca, Gianni Barlacchi, Bruno Lepri, and Luca Pappalardo. 2021.  A survey on deep learning for human mobility.  _ACM Computing Surveys (CSUR)_ 55, 1 (2021), 1–44.
  * Mai et al. (2020a) Gengchen Mai et al. 2020a.  Multi-Scale Representation Learning for Spatial Feature Distributions using Grid Cells. In _ICLR 2020_. openreview.
  * Mai et al. (2022a) Gengchen Mai, Chris Cundy, Kristy Choi, Yingjie Hu, Ni Lao, and Stefano Ermon. 2022a.  Towards a foundation model for geospatial artificial intelligence (vision paper). In _Proceedings of the 30th International Conference on Advances in Geographic Information Systems_. 1–4.
  * Mai et al. (2022b) Gengchen Mai, Yingjie Hu, Song Gao, Ling Cai, Bruno Martins, Johannes Scholz, Jing Gao, and Krzysztof Janowicz. 2022b.  Symbolic and subsymbolic GeoAI: Geospatial knowledge graphs and spatially explicit machine learning.  _Transactions in GIS_ 26, 8 (2022), 3118–3124.
  * Mai et al. (2020b) Gengchen Mai, Krzysztof Janowicz, Ling Cai, Rui Zhu, Blake Regalia, Bo Yan, Meilin Shi, and Ni Lao. 2020b.  SE-KGE: A Location-Aware Knowledge Graph Embedding Model for Geographic Question Answering and Spatial Semantic Lifting.  _Transactions in GIS_ 24, 3 (2020), 623–655.
  * Mai et al. (2022c) Gengchen Mai, Krzysztof Janowicz, Yingjie Hu, Song Gao, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. 2022c.  A review of location encoding for GeoAI: methods and applications.  _International Journal of Geographical Information Science_ 36, 4 (2022), 639–673.
  * Mai et al. (2019) Gengchen Mai, Krzysztof Janowicz, Bo Yan, Rui Zhu, Ling Cai, and Ni Lao. 2019\.  Contextual graph attention for answering logical queries over incomplete knowledge graphs. In _Proceedings of the 10th international conference on knowledge capture_. 171–178.
  * Mai et al. (2021) Gengchen Mai, Krzysztof Janowicz, Rui Zhu, Ling Cai, and Ni Lao. 2021.  Geographic question answering: challenges, uniqueness, classification, and future directions.  _AGILE: GIScience series_ 2 (2021), 8.
  * Mai et al. (2022d) Gengchen Mai, Chiyu Jiang, Weiwei Sun, Rui Zhu, Yao Xuan, Ling Cai, Krzysztof Janowicz, Stefano Ermon, and Ni Lao. 2022d.  Towards general-purpose representation learning of polygonal geometries.  _GeoInformatica_ (2022), 1–52.
  * Mendes et al. (2011) Pablo N Mendes et al. 2011\.  DBpedia spotlight: shedding light on the web of documents. In _I-Semantics 2011_.