# GeoLLM: Extracting Geospatial
Knowledge from Large Language Models

Rohin Manvi22footnotemark: 2 &Samar Khanna22footnotemark: 2 &Gengchen
Mai33footnotemark: 3 Marshall Burke22footnotemark: 2 &David
Lobell22footnotemark: 2 &Stefano Ermon22footnotemark: 2 Corresponding author,
<EMAIL_ADDRESS>UniversityUniversity of Georgia

###### Abstract

The application of machine learning (ML) in a range of geospatial tasks is
increasingly common but often relies on globally available covariates such as
satellite imagery that can either be expensive or lack predictive power. Here
we explore the question of whether the vast amounts of knowledge found in
Internet language corpora, now compressed within large language models (LLMs),
can be leveraged for geospatial prediction tasks. We first demonstrate that
LLMs embed remarkable spatial information about locations, but naively
querying LLMs using geographic coordinates alone is ineffective in predicting
key indicators like population density. We then present GeoLLM, a novel method
that can effectively extract geospatial knowledge from LLMs with auxiliary map
data from OpenStreetMap. We demonstrate the utility of our approach across
multiple tasks of central interest to the international community, including
the measurement of population density and economic livelihoods. Across these
tasks, our method demonstrates a 70% improvement in performance (measured
using Pearson’s $r^{2}$) relative to baselines that use nearest neighbors or
use information directly from the prompt, and performance equal to or
exceeding satellite-based benchmarks in the literature. With GeoLLM, we
observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51%
respectively, suggesting that the performance of our method scales well with
the size of the model and its pretraining dataset. Our experiments reveal that
LLMs are remarkably sample-efficient, rich in geospatial information, and
robust across the globe. Crucially, GeoLLM shows promise in mitigating the
limitations of existing geospatial covariates and complementing them well.

##  1 Introduction

Geospatial predictions with ML are widely used across various domains,
including poverty estimation (Jean et al., 2016; Chi et al., 2021), public
health (Nilsen et al., 2021; Areed et al., 2022; Chang et al., 2022), food
security (Nakalembe, 2018), biodiversity preservation (Mai et al., 2023c; b),
and environmental conservation (Sofaer et al., 2019). The covariates used in
these predictions include geographical coordinates, remote sensing data,
satellite imagery, human mobility data (Chang et al., 2022), and phone
metadata (Blumenstock et al., 2015; Burke et al., 2019). While having access
to quality covariates is essential, it can be challenging due to limited
spatiotemporal coverage, high costs, and accessibility barriers (Ball et al.,
2017). For example, researchers often use publicly available satellite images
to estimate socio-economic indicators like asset wealth, population density,
and infrastructure access, because they are free and globally available (Yeh
et al., 2020; Robinson et al., 2017; Head et al., 2017). However, model
predictive power can be limited due to the fact that important features may
not be visible from space.

Large language models (LLMs) have proven to be highly effective foundation
models that can be fine-tuned or prompted to perform tasks in various domains
including healthcare, education, law, finance, and scientific research
(Bommasani et al., 2021; Zhao et al., 2023). This is because LLMs have
compressed the knowledge contained in their training corpus, which includes
billions or trillions of tokens of data from the internet (Del’etang et al.,
2023). Here, we seek to understand whether LLMs possess geospatial knowledge
and explore methods to extract this knowledge to offer a novel set of
geospatial covariates that can enhance various geospatial ML tasks.

User: Describe the address "6th Avenue, Manhattan, New York" in one detailed
paragraph.

GPT-3.5: 6th Avenue, Manhattan, New York, is a bustling thoroughfare that cuts
through the heart of this iconic borough. It stretches from the southern tip
of Manhattan, where it kisses the edge of the verdant Washington Square Park,
and winds its way northward, passing through diverse neighborhoods like
Greenwich Village, Chelsea, and Midtown. The avenue is a symphony of activity,
with a constant flow of people, vehicles, and energy. Skyscrapers tower
overhead, casting long shadows that provide momentary respite from the
brilliant sunlight that often bathes the avenue. Sidewalks bustle with hurried
commuters, street vendors peddle their wares, and the distant hum of traffic
blends with the occasional blare of a taxi horn. The storefronts that line 6th
Avenue are a vibrant tapestry of businesses, ranging from cozy coffee shops
and quirky boutiques to corporate offices and renowned eateries. It’s a place
where the essence of New York City, with its unceasing vitality and diverse
culture, is palpable with every step. (a) GPT-3.5 describing an address.

Basic Prompt: Coordinates: (40.76208, -73.98042)

Population Density (On a Scale from 0.0 to 9.9):

Finetuned GPT-3.5: 5.2

Our Prompt: Coordinates: (40.76208, -73.98042)

Address: "Calyon Building, 6th Avenue, Manhattan Community Board 5, Manhattan,
New York County, City of New York, New York, United States"

Nearby Places:
"
0.6 km South-West: Theater District
0.7 km North: Columbus Circle
0.7 km East: Midtown East
0.9 km South-West: Midtown
1.0 km West: Hell’s Kitchen
1.2 km North: Lincoln Square
1.3 km South-West: Garment District
1.4 km South-East: Turtle Bay
1.4 km South: Jan Karski Corner
1.4 km South: Midtown South
"

Population Density (On a Scale from 0.0 to 9.9):

Finetuned GPT-3.5: 9.0 (b) Prompting with coordinates vs our strategy.

Figure 1: Example prompts and corresponding GPT responses. In fig. 1(a) we
show GPT-3.5 demonstrate its geospatial knowledge by asking it to describe an
address. However, in fig. 1(b) (top) prompting GPT-3.5 with just coordinates
and finetuning it on population density is insufficient. We demonstrate our
prompting strategy in fig. 1(b) (bottom) with which a finetuned GPT-3.5 is
able solve the task correctly (the expected value is 9.0).

As shown in fig. 1(a), a substantial amount of geospatial knowledge in LLMs
can be revealed simply by querying them to describe an address. However,
extracting this knowledge from LLMs is not trivial. While the most natural
interface involves using geographic coordinates like latitude and longitude to
retrieve specific location information, this often yields poor results as
shown in fig. 1(b). The inherent challenge lies in the LLMs’ capability to
understand and interpret these numeric coordinates in relation to real-world
locations, especially when the location is remote or lesser-known.

In this paper, we introduce GeoLLM, a novel method that can efficiently
extract the vast geospatial knowledge contained in LLMs by fine-tuning them on
prompts constructed with auxiliary map data from OpenStreetMap OpenStreetMap
contributors (2017). With our prompting strategy shown in fig. 1(b), we can
pinpoint a location and provide the LLM with enough spatial context
information, thereby enabling it to access and leverage its vast geospatial
knowledge. We find that including information from nearby locations in the
prompt can improve GPT-3.5’s (OpenAI, 2023b) performance by 3.3x, measured in
Pearson’s $r^{2}$, over simply providing the target location’s coordinates.

We find that popular LLM models such as GPT-3.5 and Llama 2 (Touvron et al.,
2023) can be fine-tuned to achieve state-of-the-art performance on a variety
of large-scale geospatial datasets for tasks including assessing population
density, asset wealth, mean income, women’s education and more. With GeoLLM,
GPT-3.5, Llama 2, and RoBERTa (Liu et al., 2019), show a 70%, 43%, and 13%
improvement in Pearson’s $r^{2}$ respectively over baselines that use nearest
neighbors and use information directly from the prompt, suggesting that the
models’ geospatial knowledge scales with their size and the size of their
pretraining dataset. Our experiments demonstrate LLMs’ remarkable sample
efficiency and consistency worldwide. Importantly, GeoLLM could mitigate the
limitations of existing geospatial covariates and complement them well.

We present the following main research contributions:

  1. 1.

Language models possess substantial geospatial knowledge and our method
unlocks this knowledge across a range of models and tasks.

  2. 2.

Constructing the right prompt is key to extracting geospatial knowledge. Our
ablations find that prompts constructed from map data allow the models to more
efficiently access their knowledge.

##  2 Related Work

Identifying Knowledge Pre-training instills large amounts of knowledge into
language models. Since the introduction of pre-trained language models such as
BERT (Devlin et al., 2019), many works try to quantify the amount of knowledge
they can encode (Roberts et al., 2020; Jiang et al., 2019) and see if they can
be used as knowledge bases (Petroni et al., 2019). Other works have looked for
more specific types of knowledge such as commonsense (Feldman et al., 2019),
temporal (Dhingra et al., 2021; Liu et al., 2021), biomedical (Sung et al.,
2021), numerical (Lin et al., 2020), scale (Zhang et al., 2020), and many
others. With the recent introduction of instruction-finetuned LLMs, directly
querying knowledge with prompts is a potentially simpler method. In this
paper, we focus on showing the quantity and quality of geospatial knowledge in
pre-trained language models.

Knowledge Extraction Many works aim to improve knowledge extraction from
language models by fine-tuning or prompt tuning. Works on prompt tuning
explore mining prompts from the web (Jiang et al., 2019), optimizing prompts
in the discrete space of words and tokens (Shin et al., 2020), optimizing
prompts in the continuous embedding space (Zhong et al., 2021), and using
adapters (Newman et al., 2021). However, prompts are often difficult to craft
(Qin & Eisner, 2021; Jiang et al., 2019), and are sensitive to small changes.
Fine-tuning language models for downstream tasks has shown to be an effective
and simple method to tune and elicit specific knowledge for evaluation
(Radford & Narasimhan, 2018; Dong et al., 2019). It is also shown that fine-
tuning may result in higher performance gains compared to prompt tuning
(Fichtel et al., 2021). Additionally, LoRA and quantization are shown to
greatly reduce the computational cost of fine-tuning (Hu et al., 2021;
Dettmers et al., 2023). For these reasons, we choose to extract geospatial
knowledge with fine-tuning.

NLP for Geospatial Tasks Natural language processing (NLP) has been used for
many geospatial tasks with various data sources. Hu et al. (2017) extracted
semantics relations among cities by leveraging news articles. Sheehan et al.
(2019) utilized Wikipedia to predict poverty. Xu et al. (2016) accessed
traffic by using OpenStreetMap data. Signorini et al. (2011); Paul & Dredze
(2011) utilized tweets to predict track-levels disease activities. Recently,
researchers started to explore LLMs’ capability on various geospatial tasks.
Mai et al. (2023a) demonstrated the usability of large language models on
various geospatial applications including fine-grained address recognition,
time-series dementia record forecasting, urban function prediction, and so on.
GeoGPT (Zhang et al., 2023) has been proposed as a GPT-3.5-based autonomous AI
tool that can conduct geospatial data collection, processing, and analysis in
an autonomous manner with the instruction of only natural language. However,
both works mainly relied on the pre-trained LLMs and did not explore the
possibility of fine-tuning LLMs for a geo-specific foundation model. Deng et
al. (2023) developed an LLM in geoscience, so-called K2, by fine-tuning on a
geoscience text corpus which has shown remarkable performances on various NLP
tasks in the geoscience domain. However, the K2 geoscience language foundation
model is still limited to the common NLP tasks such as question answering,
summarization, text classification, etc. In contrast, in this work, we conduct
a deep investigation into the possibility of extracting geospatial knowledge
from LLMs and whether we can use it for various real-world geospatial tasks
like predicting population density. By fine-tuning various LLMs, we
demonstrate the quantity, quality, and scalable nature of geospatial knowledge
contained in LLMs.

##  3 Method

Abstractly, we want to map geographic coordinates (latitude, longitude) and
additional features to a response variable, such as population density or
asset wealth. A naive approach would be to spatially interpolate using the
coordinates. However, this is far from ideal, especially for small sample
sizes. Features or covariates can help, but such data can be difficult to
obtain. Given this challenge, we aim to determine how much LLMs know about
these coordinates and how we can use that knowledge to better predict response
variables of interest. In this section, we describe our approach of extracting
this knowledge with the fine-tuning of language models with prompts generated
using map data.

###  3.1 Minimum Viable Geospatial Prompt

A minimum viable prompt needs to specify the location and the prediction task.
Our basic prompt consists of geographic coordinates and a description of the
task. The use of geographic coordinates is essential to extracting high-
resolution data as they are a universal and precise interface for knowledge
extraction. The description comprises the task’s name and the scale indicating
the range of possible completions. The completion, which is the ground truth
label, is also added when training an LLM. We present an example of this
simple structure in the top part of fig. 1(b).

While we find that adding the name of the task helps, more specific
information, such as the name of the dataset, generally does not help. This
suggests that basic context sufficiently assists LLMs in efficiently accessing
the knowledge in its weights.

Since the completion has to be specified through text for LLMs, we use
classification instead of regression. However, we find that a classification
task closer to regression (e.g., $0.0$ to $9.9$ as opposed to $1$ to $5$) is
beneficial, despite the fact that LLMs use softmax. If the distribution in the
original dataset already has an approximate uniform distribution, we simply
scale the values to the range from $0$ to $9.9$ and round to one decimal
place. If the dataset is not inherently uniform, we distribute the values
evenly among 100 bins to maintain a uniform distribution. Each bin is then
associated with the range of values from $0.0$ to $9.9$.

###  3.2 Prompt with Map Data

The prompt described above, however, has a major limitation. Through zero-shot
prompting, one can easily find that LLMs struggle to identify the locations of
coordinates. Not knowing where the coordinates are located would be
detrimental for geospatial tasks. This is why we focus on making sure the
prompt contains additional context that helps the model understand where the
coordinates are located. We also design our prompt to be descriptive and
useful for a range of geographical scopes such as ZIP code areas or a single
square kilometer.

Our prompt, shown in the lower part of fig. 1(b), consists of two additional
components constructed using map data:

  1. 1.

Address: A detailed reverse-geocoded description, encompassing place names
from the neighborhood level up to the country itself.

  2. 2.

Nearby Places: A list of the closest 10 locations in a 100 kilometer radius
with their respective names, distances, and directions.

We use OpenStreetMap (OpenStreetMap contributors, 2017) for our map data as it
can be queried for free using various APIs, although any map data can be used.
In our testing, map data from Google is of higher quality, but was
prohibitively expensive. We generate addresses using reverse geocoding from
Nominatim (Hoffmann, 2023) and the names and locations of nearby places from
Overpass API (Olbricht, 2023).

###  3.3 Fine-tuning and Inference with Language Models

RoBERTa $\text{RoBERTa}_{\text{BASE}}$ is a 125 million parameter encoder-only
transformer model trained on 160 GB worth of text (Liu et al., 2019). We
choose this as it and other encoder-only models like BERT (Devlin et al.,
2019) are often used for classification and regression tasks (Bommasani et
al., 2021). We fine-tune the model with all of its parameters being trainable.
It takes in the entire prompt and outputs a continuous value which we round to
the nearest bin.

Llama 2 Llama 2 7B is a 7 billion parameter decoder-only transformer model
trained on 2 trillion tokens (Touvron et al., 2023). Despite being unsuited
for regression tasks, it has the potential to perform well as it has been
trained on significantly more data. Using QLoRA (Dettmers et al., 2023), we
fine-tune Llama 2 on the prompt and completion concatenated. During inference
time, we provide it with the prompt and let it generate the three tokens
required for the completion (e.g., ”9”, ”.”, and ”9”). QLoRA allows us to
greatly reduce the computational cost and effectively fine-tune with just 33
million trainable parameters. However, since it also quantizes the frozen
model to 4-bit, this comes with the potential cost of diluting the knowledge
contained in the weights. We find that fine-tuning for 4 epochs with QLoRA and
various regularization techniques works well. Interestingly, the model’s
performance still improves even when the loss on the validation set increases
by the fourth epoch. This suggests that the model is less prone to overfitting
on the prediction tasks compared to the prompts.

GPT-3.5 GPT-3.5 (gpt-3.5-turbo-0613) (OpenAI, 2023b) is a chat based LLM. It
is the successor of GPT-3 which is a decoder-only transformer model with 175
billion parameters (Brown et al., 2020). GPT-3.5 shows promise in containing a
substantial amount of knowledge (Bian et al., 2023). We fine-tune GPT-3.5
through OpenAI’s fine-tuning API (OpenAI, 2023a). Similar to Llama 2, we
provide it the prompt and let it generate the three tokens required for the
completion. We find that 4, 3, and 2 epochs work well for 100, 1,000, and
10,000 training samples respectively. It is worth noting that the specific
fine-tuning procedures and internal architecture of GPT-3.5 are not publicly
disclosed by OpenAI.

##  4 Experiments

We construct a comprehensive benchmark that encompasses a wide range of
geospatial prediction tasks. This benchmark includes various domains and
geographies, ranging from women’s education levels in developing countries to
housing prices in the US. We develop a robust set of baselines to demonstrate
the performance attainable using the information directly in the prompt.
Finally, we present our results and an ablation study.

Task | Source | Samples | GPT-3.5 | Llama 2 | RoBERTa | XGBoost-FT | XGBoost | k-NN
---|---|---|---|---|---|---|---|---
Population | WorldPop | 10,000 | $0.776$ | $0.733$ | $0.604$ | $0.528$ | $0.475$ | $0.359$
1,000 | $0.733$ | $0.630$ | $0.413$ | $0.342$ | $0.322$ | $0.112$
100 | $0.608$ | $0.432$ | $0.004$ | $0.135$ | $0.189$ | $0.018$
Asset Wealth | DHS | 10,000 | $0.752$ | $0.735$ | $0.692$ | $0.641$ | $0.634$ | $0.589$
1,000 | $0.714$ | $0.618$ | $0.554$ | $0.461$ | $0.479$ | $0.437$
Women Edu | DHS | 10,000 | $0.639$ | $0.614$ | $0.576$ | $0.551$ | $0.550$ | $0.542$
1,000 | $0.596$ | $0.520$ | $0.471$ | $0.356$ | $0.397$ | $0.399$
Sanitation | DHS | 10,000 | $0.646$ | $0.633$ | $0.617$ | $0.605$ | $0.571$ | $0.573$
1,000 | $0.636$ | $0.513$ | $0.491$ | $0.426$ | $0.423$ | $0.429$
Women BMI | DHS | 10,000 | $0.632$ | $0.631$ | $0.602$ | $0.587$ | $0.585$ | $0.581$
1,000 | $0.580$ | $0.557$ | $0.503$ | $0.425$ | $0.460$ | $0.499$
Population | USCB | 1,000 | $0.700$ | $0.502$ | $0.342$ | $0.240$ | $0.265$ | $0.207$
100 | $0.616$ | $0.376$ | $0.002$ | $0.125$ | $0.150$ | $0.084$
Mean Income | USCB | 1,000 | $0.552$ | $0.428$ | $0.211$ | $0.150$ | $0.181$ | $0.221$
100 | $0.460$ | $0.294$ | $0.006$ | $0.031$ | $0.042$ | $0.100$
Hispanic Ratio | USCB | 1,000 | $0.741$ | $0.651$ | $0.548$ | $0.501$ | $0.561$ | $0.567$
100 | $0.644$ | $0.317$ | $0.108$ | $0.192$ | $0.230$ | $0.400$
Home Value | Zillow | 1,000 | $0.874$ | $0.743$ | $0.496$ | $0.461$ | $0.503$ | $0.590$
100 | $0.739$ | $0.431$ | $0.000$ | $0.126$ | $0.272$ | $0.403$
Table 1: Pearson’s $r^{2}$ for all models across all tasks and training sample
sizes. Figure 2: Plots of absolute error comparing the best baselines and
GPT-3.5 on tasks from each source with 1,000 samples. We also provide high-
resolution plots from various locations around the world for the population
density task from WorldPop. We show that GeoLLM not only outperforms baselines
on a variety of tasks (table 1) but also demonstrates remarkable geographic
consistency across the world.

###  4.1 Tasks and Sources

WorldPop WorldPop (Tatem, 2017) is a research group focused on providing
spatial demographic data for the use of health and development applications,
including addressing the Sustainable Development Goals. We use their
population counts dataset that covers the whole world for the population
density task. More specifically, we use their unconstrained global mosaic for
2020 that has a resolution of 30 arc (approximately 1km at the equator)
(WorldPop & CIESIN, 2018). To ensure a comprehensive representation of
populations, we employed importance sampling, weighted by population. This
allows us to capture a wide range of populations; without it, our samples
would largely consist of sparsely populated areas. To construct a learning
curve for this task, we use training sample sizes of 100, 1,000, and 10,000.

DHS SustainBench (Yeh et al., 2021) includes tasks derived from survey data
from the Demographic and Health Surveys (DHS) program (DHS, 2023). These
surveys constitute nationally representative household-level data on assets,
housing conditions, and education levels, among other attributes across 48
countries. While DHS surveys provide household-level data, SustainBench
summarizes the survey data into “cluster-level” labels, where a “cluster”
roughly corresponds to a village or local community. The geocoordinates are
“jittered” randomly up to 2km for urban clusters and 10km for rural clusters
to protect survey participant privacy. From their datasets, we use asset
wealth index, women educational attainment, sanitation index, and women BMI.
Since their datasets cover a wide scope, we use training sample sizes of 1,000
and 10,000.

USCB The United States Census Bureau (USCB) (US, 2023) is responsible for
producing data about the American people and economy. In addition to the
decennial census, the Census Bureau continually conducts annual demographics
surveys through the American Community Survey (ACS) program. From these two
sources, we get tasks for population, mean income, and Hispanic to Non-
Hispanic ratio by what they call ZIP Code Tabulation Areas. Since their data
only covers the US, we use training sample sizes of 100 and 1,000.

Zillow Zillow provides the Zillow Home Value Index (ZHVI) (ZR, 2023): A
measure of the typical home value and market changes across a given region and
housing type. It reflects the typical value for homes in the 35th to 65th
percentile range. We use the smoothed, seasonally adjusted measure that they
provide. Like with the USCB data, we also use this data on a ZIP code level.
This serves as our home value task. Since this dataset also only covers the
US, we use training sample sizes of 100 and 1,000.

When explicit coordinates are not given in the datasets, we approximate them
by determining the center of relevant areas. For instance, we approximate the
center of each of the ZIP code area so that we can generate prompts for the
tasks from USCB or Zillow. We evaluate performance using the squared Pearson
correlation coefficient $r^{2}$ between the predictions and labels so that
comparisons can be made with previous literature (Perez et al., 2017; Jean et
al., 2016; Yeh et al., 2020; 2021). We use 2,000 samples for our test and
validation sets across all tasks. We split the data into training, test, and
validation partitions early in the process before sampling different sizes. We
also explicitly check again for any overlap between the train and test samples
before fine-tuning or training.

###  4.2 Baselines

The purpose of our three baselines is to demonstrate the performance
attainable using only the information in the prompt. Pre-trained language
models can surpass the baselines only if they effectively use the knowledge
embedded in their weights. This helps us understand the extent of knowledge
encapsulated within pre-trained language models.

k-NN Our simplest baseline is k-NN. It takes the mean of the 5 closest
neighbors. This is a reasonable baseline as physical distance is very relevant
for geospatial tasks.

XGBoost Our second baseline uses XGBoost (Chen & Guestrin, 2016). This
baseline incorporates all numerical and categorical data from the prompt,
including coordinates, distances, and directions. It could outperform k-NN
since it leverages not just the coordinates, but also the distance and
direction data. We established this baseline believing that the direction and
distance information could be covariates for some of the tasks we use.

XGBoost-FT Our third baseline, XGBoost-FT, leverages both XGBoost and fastText
(Joulin et al., 2016). This baseline uses all information contained in the
prompt. In addition to using the coordinates, distances, and directions, it
also uses embeddings generated from the address and the names of the nearby
places. We use fastText to learn sentence embeddings on any given dataset
using strings created by concatenating the address and the names of the nearby
places.

All baselines treat the tasks as a regression problem. Their output is rounded
to the nearest bin ($0.0$ to $9.9$) for classification. Any improvements over
these baselines would likely come from knowledge about the address or list of
places.

###  4.3 Performance on Tasks

From the results presented in table 1, it is evident that LLMs are the best-
performing models. Among them, GPT-3.5 stands out as the best-performing LLM.
Its Pearson’s $r^{2}$ ranges from 0.46 to 0.74, 0.55 to 0.87, and 0.63 to 0.78
for sample sizes of 100, 1,000, and 10,000, respectively. Improvements over
the best baselines range from a Pearson’s $r^{2}$ of 0.24 to 0.47, 0.08 to
0.44, and 0.04 to 0.25 for those same sample sizes. While these variations
indicate that the tasks differ in difficulty, LLMs generally outperform all
the baselines. This suggests that LLMs possess a substantial amount of
geospatial knowledge.

Not only does GPT-3.5 outperform all other models on every test, but its
performance is also relatively consistent across tasks and sample sizes. This
shows that GPT-3.5 is resilient to the size of the areas of prediction (e.g.,
square kilometer vs ZIP code area), and any added noise (e.g., ”jittered”
coordinates). Llama 2 also does better than all baselines for 18 out of the 19
total tests and consistently does better than RoBERTa. RoBERTa consistently
does better than all baselines with 10,000 training samples, but struggles at
lower sample sizes.

All models experience a significant drop in performance when the sample size
is reduced to 100. However, GPT-3.5 and Llama 2 retain a much more acceptable
level of performance compared to the others, emphasizing their sample
efficiency. Specifically, with 100 samples, GPT-3.5 does 3.1 times better than
the best baseline and Llama 2 does 1.8 times better.

GPT-3.5 and Llama 2 do especially well with the population tasks from WorldPop
and the USCB compared to the baselines. GPT-3.5 is also especially impressive
with the home value task from Zillow with a Pearon’s $r^{2}$ of up to 0.87.
However, the difference in performance between the models is less pronounced
for the tasks from the DHS. This might be due to the noise that is added when
the coordinates for these tasks are “jittered” up to 5 kilometers. With the
added noise, it is potentially more difficult to achieve a high Pearson’s
$r^{2}$.

As shown in fig. 4 GPT-3.5, Llama 2, and RoBERTa, perform 70%, 43%, and 13%
better on average than the best baseline (XGBoost) respectively with 1,000
samples, indicating that the method scales well. Fig. 4 again shows that the
sample efficiencies of Llama 2 and GPT-3.5 are exceptional, especially when
making predictions on a global scale. Note that with larger sample sizes the
gaps in performance will decrease as the physical distances between the
training coordinates and test coordinates become smaller.

It is visually clear in fig. 2 that GPT-3.5 does substantially better than the
best baselines for each task. It also appears that GPT-3.5 generally only
struggles where the baselines struggle, suggesting that those areas might just
be difficult to estimate rather than it being a specific failing of the model.
Notably, GPT-3.5 shows has impressive geographic consistency.

GPT-3.5Llama 2RoBERTaXGBoost-FTXGBoostk-
NN$0$$0.2$$0.4$$0.6$$0.8$$0.68$$0.57$$0.45$$0.37$$0.4$$0.39$Mean $r^{2}$
performance Figure 3: Mean Pearson’s $r^{2}$ for each model across all tasks
at 1,000 training samples

1001,00010,000$0$$0.1$$0.2$$0.3$$0.4$$0.5$$0.6$$0.7$$0.8$Training Sample
Size$r^{2}$k-NNXGBoostXGBoost-FTRoBERTaLlama 2GPT-3.5 Figure 4: Learning
curves for population density task from WorldPop.

###  4.4 Ablations on the Prompt

The ablation study shown in table 2 demonstrates the importance of using a
highly descriptive prompt that helps the model understand where the
coordinates are located and efficiently access the knowledge contained in its
weights. It also suggests that all components of the prompt contribute to the
overall performance of the models. In addition to performing better than the
other language models, GPT-3.5 is more resilient to the removal of the map
data. This is evident from the fact that prompt without map data is completely
ineffective for Llama 2 and RoBERTa. This suggests that GPT-3.5 can more
effectively access its knowledge.

Ablation | GPT-3.5 | Llama 2 | RoBERTa
---|---|---|---
Whole prompt | $0.733$ | $0.630$ | $0.413$
Removed address | $0.678$ | $0.514$ | $0.375$
Removed nearby places | $0.579$ | $0.473$ | $0.281$
Removed address and nearby places | $0.219$ | $0.002$ | $0.001$
Table 2: Pearson’s $r^{2}$ for ablations with the population density task from
WorldPop and 1,000 training samples.

##  5 Discussion

LLMs not only match but also exceed the performance of methods that use
satellite imagery. Satellite-based methods achieve Pearson’s $r^{2}$ of 0.56
(Jean et al., 2016), 0.71 (Perez et al., 2017), and 0.67 (Yeh et al., 2020) in
predicting asset wealth in Africa. With 10,000 training samples (approximately
3,700 of which are from Africa), a fraction of the over 87,000 data points
available from the DHS, GPT-3.5 achieves an $r^{2}$ of 0.72 when evaluated in
Africa.

The performance of the LLMs is particularly notable considering an inherent
disadvantage they have relative to the baselines and RoBERTa. Generating
multiple tokens with softmax is far less intuitive than regression which the
other models use. Despite this, there are substantial improvements in
performance and sample efficiency with the LLMs.

It also appears that the significant increase in performance is not simply due
to the number of parameters being trained during fine-tuning. With QLoRA,
Llama 2 has 33 million trainable parameters as opposed to RoBERTa’s 125
million trainable parameters. This suggests that the performance increase is
due to the knowledge contained in Llama 2’s frozen weights rather than the
parameters that are being trained.

This method’s performance appears to scale with both the size of the model’s
pretraining dataset and its capacity for data compression, a capability
influenced by the model’s size. As both of these factors generally enhance
language models (Kaplan et al., 2020), our method likely scales with the
general capability of the model as well. For example, GPT-4 (OpenAI, 2023c)
would likely outperform GPT-3.5, which is especially apparent when prompting
the models to describe an address as shown in fig. 1(a).

Since LLMs are trained on internet data, they likely have biases similar to
the ones present in this data. One could even potentially use our method to
better understand the biases of LLMs and their training corpora. Surprisingly,
the performance across countries and continents appears to be consistent with
the LLMs, as evident from the maps of absolute errors shown in fig. 2.

This work lays the foundation for the future use of LLMs for geospatial tasks.
Additional dimensions could extend the applicability of our method. For
instance, one could augment prompts with names of amenities or roads to enrich
the geospatial context and allow for even higher-resolution knowledge
extraction. Temporal data such as dates could be integrated into the prompts
for making forecasts. Multimodal LLMs could potentially also leverage
satellite imagery or street view images. Overall, there are likely many ways
to extend GeoLLM.

##  6 Conclusion

In this work, we introduced GeoLLM, a novel method that can efficiently
extract the vast geospatial knowledge contained in LLMs by fine-tuning them on
prompts constructed with auxiliary map data from OpenStreetMap. We found that
including information from nearby locations in the prompt greatly improves
LLMs’ performance. We demonstrated the utility of our approach across various
tasks from multiple large-scale real-world datasets. Our method demonstrated a
70% improvement in Pearson’s $r^{2}$ over traditional baselines including k-NN
and XGBoost, exceeding the performance of satellite-based methods. With
GeoLLM, we observed that GPT-3.5 outperforms Llama 2 by 19% and RoBERTa by
51%, suggesting that the performance of our method scales well with the size
of the model and its pretraining dataset. Our simple method revealed that LLMs
are sample-efficient, rich in geospatial information, and robust across the
globe. Crucially, GeoLLM shows promise in substantially mitigating the
limitations of traditional geospatial covariates.

## References

  * DHS (2023) The demographic and health surveys (dhs) program, 2023.  URL http://www.dhsprogram.com.
  * US (2023) Census bureau tables, 2023.  URL https://data.census.gov/table.
  * ZR (2023) Housing data, 2023.  URL https://www.zillow.com/research/data/.
  * Areed et al. (2022) Wala Draidi Areed, Aiden Price, Kathryn D Arnett, and Kerrie Lee Mengersen.  Spatial statistical machine learning models to assess the relationship between development vulnerabilities and educational factors in children in queensland, australia.  _BMC Public Health_ , 22, 2022.  URL https://api.semanticscholar.org/CorpusID:254071901.
  * Ball et al. (2017) John E. Ball, Derek T. Anderson, and Chee Seng Chan.  Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community.  _Journal of Applied Remote Sensing_ , 11, 2017.  URL https://api.semanticscholar.org/CorpusID:7069900.
  * Bian et al. (2023) Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He.  Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models.  _ArXiv_ , abs/2303.16421, 2023.  URL https://api.semanticscholar.org/CorpusID:257804619.
  * Blumenstock et al. (2015) Joshua Evan Blumenstock, Gabriel Cadamuro, and Robert On.  Predicting poverty and wealth from mobile phone metadata.  _Science_ , 350:1073 – 1076, 2015.  URL https://api.semanticscholar.org/CorpusID:206638993.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _ArXiv_ , abs/2005.14165, 2020.  URL https://api.semanticscholar.org/CorpusID:218971783.
  * Burke et al. (2019) M. Burke, Anne Driscoll, D. Lobell, and Stefano Ermon.  Using satellite imagery to understand and promote sustainable development.  _Science_ , 371, 2019.  URL https://api.semanticscholar.org/CorpusID:222305990.
  * Chang et al. (2022) Ting Chang, Yingjie Hu, Dane Taylor, and Brian M Quigley.  The role of alcohol outlet visits derived from mobile phone location data in enhancing domestic violence prediction at the neighborhood level.  _Health & Place_, 73:102736, 2022.
  * Chen & Guestrin (2016) Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016.  URL https://api.semanticscholar.org/CorpusID:4650265.
  * Chi et al. (2021) Guanghua Chi, Han Fang, Sourav Chatterjee, and Joshua Evan Blumenstock.  Microestimates of wealth for all low- and middle-income countries.  _Proceedings of the National Academy of Sciences of the United States of America_ , 119, 2021.  URL https://api.semanticscholar.org/CorpusID:233289683.
  * Del’etang et al. (2023) Gr’egoire Del’etang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Wenliang Kevin Li, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness.  Language modeling is compression.  2023\.  URL https://api.semanticscholar.org/CorpusID:262054258.
  * Deng et al. (2023) Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Le Zhou, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al.  Learning a foundation language model for geoscience knowledge understanding and utilization.  _arXiv preprint arXiv:2306.05064_ , 2023.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms.  _ArXiv_ , abs/2305.14314, 2023.  URL https://api.semanticscholar.org/CorpusID:258841328.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805, 2019.  URL https://api.semanticscholar.org/CorpusID:52967399.
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273, 2021.  URL https://api.semanticscholar.org/CorpusID:235669861.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, M. Zhou, and Hsiao-Wuen Hon.  Unified language model pre-training for natural language understanding and generation.  In _Neural Information Processing Systems_ , 2019.  URL https://api.semanticscholar.org/CorpusID:147704286.
  * Feldman et al. (2019) Joshua Feldman, Joe Davison, and Alexander M. Rush.  Commonsense knowledge mining from pretrained models.  _ArXiv_ , abs/1909.00505, 2019.  URL https://api.semanticscholar.org/CorpusID:202541043.
  * Fichtel et al. (2021) Leandra Fichtel, Jan-Christoph Kalo, and Wolf-Tilo Balke.  Prompt tuning or fine-tuning - investigating relational knowledge in pre-trained language models.  In _3rd Conference on Automated Knowledge Base Construction_ , 2021\.  URL https://openreview.net/forum?id=o7sMlpr9yBW.
  * Head et al. (2017) Andrew Head, Mélanie Manguin, Nhat Tran, and Joshua Evan Blumenstock.  Can human development be measured with satellite imagery?  _Proceedings of the Ninth International Conference on Information and Communication Technologies and Development_ , 2017.  URL https://api.semanticscholar.org/CorpusID:1763100.
  * Hoffmann (2023) Sarah Hoffmann.  Nominatim, 2023.  URL https://nominatim.org.
  * Hu et al. (2021) J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _ArXiv_ , abs/2106.09685, 2021.  URL https://api.semanticscholar.org/CorpusID:235458009.
  * Hu et al. (2017) Yingjie Hu, Xinyue Ye, and Shih-Lung Shaw.  Extracting and analyzing semantic relatedness between cities using news articles.  _International Journal of Geographical Information Science_ , 31(12):2427–2451, 2017.
  * Jean et al. (2016) Neal Jean, M. Burke, Sang Michael Xie, W. Matthew Davis, D. Lobell, and Stefano Ermon.  Combining satellite imagery and machine learning to predict poverty.  _Science_ , 353:790 – 794, 2016.  URL https://api.semanticscholar.org/CorpusID:16154009.
  * Jiang et al. (2019) Zhengbao Jiang, Frank F. Xu, J. Araki, and Graham Neubig.  How can we know what language models know?  _Transactions of the Association for Computational Linguistics_ , 8:423–438, 2019.  URL https://api.semanticscholar.org/CorpusID:208513249.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov.  Bag of tricks for efficient text classification.  _arXiv preprint arXiv:1607.01759_ , 2016.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei.  Scaling laws for neural language models.  _ArXiv_ , abs/2001.08361, 2020.  URL https://api.semanticscholar.org/CorpusID:210861095.
  * Lin et al. (2020) Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren.  Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models.  In _Conference on Empirical Methods in Natural Language Processing_ , 2020.  URL https://api.semanticscholar.org/CorpusID:218486812.
  * Liu et al. (2021) Leo Z. Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith.  Probing across time: What does roberta know and when?  In _Conference on Empirical Methods in Natural Language Processing_ , 2021.  URL https://api.semanticscholar.org/CorpusID:233289478.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692, 2019.  URL https://api.semanticscholar.org/CorpusID:198953378.
  * Mai et al. (2023a) Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, et al.  On the opportunities and challenges of foundation models for geospatial artificial intelligence.  _arXiv preprint arXiv:2304.06798_ , 2023a.
  * Mai et al. (2023b) Gengchen Mai, Ni Lao, Yutong He, Jiaming Song, and Stefano Ermon.  Csp: Self-supervised contrastive spatial pre-training for geospatial-visual representations.  In _the Fortieth International Conference on Machine Learning (ICML 2023)_ , 2023b.
  * Mai et al. (2023c) Gengchen Mai, Yao Xuan, Wenyun Zuo, Yutong He, Jiaming Song, Stefano Ermon, Krzysztof Janowicz, and Ni Lao.  Sphere2vec: A general-purpose location representation learning over a spherical surface for large-scale geospatial predictions.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 202:439–462, 2023c.
  * Nakalembe (2018) Catherine Nakalembe.  Characterizing agricultural drought in the karamoja subregion of uganda with meteorological and satellite-based indices.  _Natural Hazards_ , 91:837–862, 2018.  URL https://api.semanticscholar.org/CorpusID:133875170.
  * Newman et al. (2021) Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani.  P-adapters: Robustly extracting factual information from language models with diverse prompts.  _ArXiv_ , abs/2110.07280, 2021.  URL https://api.semanticscholar.org/CorpusID:238856973.
  * Nilsen et al. (2021) Kristine Nilsen, Natalia Tejedor-Garavito, Douglas R. Leasure, Chigozie Edson Utazi, Corrine Warren Ruktanonchai, Adelle Wigley, Claire A. Dooley, Zoe Matthews, and Andrew J. Tatem.  A review of geospatial methods for population estimation and their use in constructing reproductive, maternal, newborn, child and adolescent health service indicators.  _BMC Health Services Research_ , 21, 2021.  URL https://api.semanticscholar.org/CorpusID:237487421.
  * Olbricht (2023) Roland Olbricht.  Overpass api, 2023.  URL https://wiki.openstreetmap.org/wiki/Overpass_API.
  * OpenAI (2023a) OpenAI.  Gpt-3.5 turbo: Fine-tuning and api updates, 2023a.  URL https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates.
  * OpenAI (2023b) OpenAI.  Introducing chatgpt, 2023b.  URL https://openai.com/blog/chatgpt.
  * OpenAI (2023c) OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023c.  URL https://api.semanticscholar.org/CorpusID:257532815.
  * OpenStreetMap contributors (2017) OpenStreetMap contributors.  Planet dump retrieved from https://planet.osm.org .  https://www.openstreetmap.org, 2017.
  * Paul & Dredze (2011) Michael J. Paul and Mark Dredze.  You are what you tweet: Analyzing twitter for public health.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 2011.  URL https://api.semanticscholar.org/CorpusID:9270435.
  * Perez et al. (2017) Anthony Perez, Christopher Yeh, George Azzari, M. Burke, D. Lobell, and Stefano Ermon.  Poverty prediction with public landsat 7 satellite imagery and machine learning.  _ArXiv_ , abs/1711.03654, 2017.  URL https://api.semanticscholar.org/CorpusID:23748178.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel.  Language models as knowledge bases?  _ArXiv_ , abs/1909.01066, 2019.  URL https://api.semanticscholar.org/CorpusID:202539551.
  * Qin & Eisner (2021) Guanghui Qin and Jas’ Eisner.  Learning how to ask: Querying lms with mixtures of soft prompts.  _ArXiv_ , abs/2104.06599, 2021.  URL https://api.semanticscholar.org/CorpusID:233231453.
  * Radford & Narasimhan (2018) Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training.  2018\.  URL https://api.semanticscholar.org/CorpusID:49313245.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam M. Shazeer.  How much knowledge can you pack into the parameters of a language model?  In _Conference on Empirical Methods in Natural Language Processing_ , 2020.  URL https://api.semanticscholar.org/CorpusID:211205183.
  * Robinson et al. (2017) Caleb Robinson, Fred Hohman, and Bistra N. Dilkina.  A deep learning approach for population estimation from satellite imagery.  _Proceedings of the 1st ACM SIGSPATIAL Workshop on Geospatial Humanities_ , 2017.  URL https://api.semanticscholar.org/CorpusID:12713975.
  * Sheehan et al. (2019) Evan Sheehan, Chenlin Meng, Matthew Sze Piao. Tan, Burak Uzkent, Neal Jean, D. Lobell, M. Burke, and Stefano Ermon.  Predicting economic development using geolocated wikipedia articles.  _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019.  URL https://api.semanticscholar.org/CorpusID:146120697.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Eliciting knowledge from language models using automatically generated prompts.  In _Conference on Empirical Methods in Natural Language Processing_ , 2020.  URL https://api.semanticscholar.org/CorpusID:226222232.
  * Signorini et al. (2011) Alessio Signorini, Alberto Maria Segre, and Philip M. Polgreen.  The use of twitter to track levels of disease activity and public concern in the u.s. during the influenza a h1n1 pandemic.  _PLoS ONE_ , 6, 2011.  URL https://api.semanticscholar.org/CorpusID:6808103.
  * Sofaer et al. (2019) Helen R. Sofaer, Catherine S. Jarnevich, Ian S. Pearse, Regan L. Smyth, Stephanie Auer, Gericke Cook, Thomas C. Edwards, Gerald F. Guala, Timothy G. Howard, Jeffrey T. Morisette, and Healy Hamilton.  Development and delivery of species distribution models to inform decision-making.  _BioScience_ , 2019.  URL https://api.semanticscholar.org/CorpusID:195384775.
  * Sung et al. (2021) Mujeen Sung, Jinhyuk Lee, Sean S. Yi, Minji Jeon, Sungdong Kim, and Jaewoo Kang.  Can language models be biomedical knowledge bases?  _ArXiv_ , abs/2109.07154, 2021.  URL https://api.semanticscholar.org/CorpusID:237513360.
  * Tatem (2017) Andrew J. Tatem.  Worldpop, open data for spatial demography.  _Scientific Data_ , 4, 2017.  URL https://api.semanticscholar.org/CorpusID:3544507.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv_ , abs/2307.09288, 2023.  URL https://api.semanticscholar.org/CorpusID:259950998.
  * WorldPop & CIESIN (2018) WorldPop and Columbia University CIESIN.  Global high resolution population denominators project - funded by the bill and melinda gates foundation, 2018.  URL https://dx.doi.org/10.5258/SOTON/WP00647.  www.worldpop.org - School of Geography and Environmental Science, University of Southampton; Department of Geography and Geosciences, University of Louisville; Departement de Geographie, Universite de Namur.
  * Xu et al. (2016) Frank F. Xu, Bill Yuchen Lin, Qi Lu, Yifei Huang, and Kenny Q. Zhu.  Cross-region traffic prediction for china on openstreetmap.  _Proceedings of the 9th ACM SIGSPATIAL International Workshop on Computational Transportation Science_ , 2016.  URL https://api.semanticscholar.org/CorpusID:15751676.
  * Yeh et al. (2020) Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, D. Lobell, Stefano Ermon, and M. Burke.  Using publicly available satellite imagery and deep learning to understand economic well-being in africa.  _Nature Communications_ , 11, 2020.  URL https://api.semanticscholar.org/CorpusID:218773287.
  * Yeh et al. (2021) Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, D. Lobell, and Stefano Ermon.  Sustainbench: Benchmarks for monitoring the sustainable development goals with machine learning.  _ArXiv_ , abs/2111.04724, 2021.  URL https://api.semanticscholar.org/CorpusID:243847865.
  * Zhang et al. (2020) Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth.  Do language embeddings capture scales?  In _BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , 2020.  URL https://api.semanticscholar.org/CorpusID:222290454.
  * Zhang et al. (2023) Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu.  Geogpt: Understanding and processing geospatial tasks through an autonomous gpt.  _arXiv preprint arXiv:2307.07930_ , 2023.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen.  A survey of large language models.  _ArXiv_ , abs/2303.18223, 2023.  URL https://api.semanticscholar.org/CorpusID:257900969.
  * Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen.  Factual probing is [mask]: Learning vs. learning to recall.  In _North American Chapter of the Association for Computational Linguistics_ , 2021.  URL https://api.semanticscholar.org/CorpusID:233210199.


# Transformer Masked Autoencoders for Next-Generation Wireless Communications:
Architecture and Opportunities ††thanks: A. Zayat, M. A. Hasabelnaby, and A.
Chaaban are with the School of Engineering, University of British Columbia,
Kelowna, BC V1V1V7, Canada. ††thanks: M. Obeed is with the Systems and
Computer Engineering Department, Carleton University, Ottawa, ON K1S 5B6,
Canada.

Abdullah Zayat, Mahmoud A. Hasabelnaby, Graduate Student Member, IEEE, Mohanad
Obeed, Member, IEEE, Anas Chaaban, Senior Member, IEEE

###### Abstract

Next-generation communication networks are expected to exploit recent advances
in data science and cutting-edge communications technologies to improve the
utilization of the available communications resources. In this article, we
introduce an emerging deep learning (DL) architecture, the transformer-masked
autoencoder (TMAE), and discuss its potential in next-generation wireless
networks. We discuss the limitations of current DL techniques in meeting the
requirements of 5G and beyond 5G networks, and how the TMAE differs from the
classical DL techniques can potentially address several wireless communication
problems. We highlight various areas in next-generation mobile networks which
can be addressed using a TMAE, including source and channel coding,
estimation, and security. Furthermore, we demonstrate a case study showing how
a TMAE can improve data compression performance and complexity compared to
existing schemes. Finally, we discuss key challenges and open future research
directions for deploying the TMAE in intelligent next-generation mobile
networks.

###### Index Terms:

6G, 5G, convolutional neural networks, deep learning, wireless communication,
recurrent neural networks, transformer, masked autoencoder.

##  I Introduction

Next-generation (NG) mobile networks are increasingly calling for intelligent
architectures that support massive connectivity, ultra-low latency, ultra-high
reliability, high-quality of experience, high spectral and energy efficiency,
and lower deployment costs [1]. One way to meet these stringent requirements
is to rethink traditional communication techniques by exploiting recent
advances in artificial intelligence.

Traditionally, functions such as waveform design, channel estimation,
interference mitigation, and error detection and correction are developed
based on theoretical models and assumptions. This traditional approach is not
capable of adapting to new challenges introduced by emerging technologies. For
instance, the pilot-based channel estimation technique, while efficient for
MIMO systems with a few antennas and low-mobility users, is not efficient for
massive multiple-input multiple-output (MIMO) systems or high-mobility users.
Additionally, the plethora of communication protocols, technologies, and
services that have been introduced to support the growing demand and diversity
of use cases make it increasingly difficult to mathematically model wireless
networks. As such, optimizing communication schemes using mathematical models
becomes extremely challenging and computationally complex, particularly with
the integration of demanding 6G applications and services, such as the
Metaverse, ubiquitous Extended Reality (XR), intelligent connected robotics,
large-scale intelligent reconfigurable surfaces (IRSs), and ultra-massive MIMO
(UM-MIMO) networks.

Due to this, the use of deep learning (DL) approaches has been proposed to
solve wireless communications challenges due to their ability to adapt to
dynamic environments, approximate complex models, and utilize data to improve
performance [2]. Transformer-enabled DL, initially proposed for natural
language processing (NLP) tasks [3], opens the door for further advances in
this area. The main advantage of transformers is their superior ability to
learn complex dependencies between input features compared to classical deep
neural networks (DNN) [4]. The goal of this paper is to discuss a transformer-
based architecture, the transformer masked autoencoder (TMAE), and its
application in communications. Particularly, we start by highlighting some
limitations of existing DNNs (Sec. II), then we explain the general
transformer and the MAE architectures (Sec. III). Next, we present a use case
where a TMAE enhances data compression by exploiting semantics, which can
significantly improve achievable rates in wireless networks (Sec. IV).
Finally, we discuss opportunities for improving wireless communications using
the TMAE (Sec. V) and summarize the takeaway message of the paper (Sec. VI).

##  II Classical DNN Limitations in NG Networks

Recent advances in DL opened the possibility for designing intelligent mobile
networks that learn to operate optimally using massive amounts of data [2],
thus overcoming mathematical modeling and computational complexity challenges
[5]. Although DL offers advantages over mathematical models, it is not free of
limitations. Some common DNNs are discussed next, followed by their
limitations in NG networks.

###  II-A Common DNN Architectures

The most commonly used DNN architectures in wireless communications include
the following:

  * •

Multi-Layer Perceptions (MLP): An MLP is a feed-forward neural network (NN)
that consists of at least three layers of fully-connected nodes: an input
layer, a hidden layer, and an output layer. MLPs have been proposed to tackle
various mobile network problems, such as beamforming and channel estimation
[5].

  * •

Convolutional Neural Networks (CNN): CNNs replace fully-connected layers with
locally connected kernels that capture local correlations in data. This
reduces the number of model parameters which simplifies training and reduces
the risk of overfitting. As a result, CNNs often outperform MLPs in many
applications such as computer vision (CV). CNNs have also been used in many
wireless communications applications such as semantic communications [2],
beamforming, channel estimation, and channel state information (CSI) feedback
[5].

  * •

Recurrent Neural Networks (RNN): The main difference between a typical NN (MLP
or CNN) and an RNN is that an RNN has feedback connections in addition to
feed-forward connections. These connections give RNNs memory of previous
inputs/outputs, which is useful for sequential processing and helps in
exploiting local and global correlations. This makes them ideal for
applications with time-series data [5].

  * •

Long Short-Term Memory (LSTM) Networks: Although RNNs have memory, their
memory is short. As a special type of RNNs, LSTM networks feature gated memory
cells which extends their memory to longer sequences. Due to this, LSTM
networks have been used for channel estimation in channels with memory [5].

###  II-B Limitations in NG Networks

The aforementioned NNs can be employed to build encoding and decoding layers
(an autoencoder), that produce a different representation of data and
reconstruct it from the new representation, respectively. This can then be
used to realize physical layer processing tasks such as signal design, channel
estimation, CSI feedback, modulation, and coding [5].

However, classical DNNs encounter limitations in fully meeting the demands of
NG networks. MLPs have limited ability to extract deep features from raw data,
and their performance on sequential data is poor. This leads to challenges in
generalizing and transferring learned knowledge between different scenarios,
hindering their ability to seamlessly adapt and perform in a dynamically
changing environment which is common in wireless networks. While CNNs, RNNs,
and LSTM networks are more adaptable spatially/temporally to local/short-term
changes, they have limited capability to effectively exploit global/long-term
dependencies in sequential data, which is crucial for capturing the intricate
patterns and dynamics inherent in NG wireless networks. In addition to this,
training recurrent networks (RNN and LSTM) suffers from challenges related to
convergence, vanishing gradients, and parallelization, which limits their
usefulness in latency-sensitive applications.

These limitations underscore the need for innovative enhancements, alternative
architectures, and hybrid approaches to effectively address the evolving
requirements of NG networks. Recently, attention-based DL realized using
transformer networks was proposed and shown to achieve remarkable performance
gains in various CV and NLP applications compared to classical DL [3]. Next,
we present the basic architecture of transformers and discuss their potential
compared to classical DNNs. The potential of the TMAE, a transformer-based
autoencoder, for NG networks, is discussed afterward.

##  III Potential of Transformer-Based NNs

A transformer is an NN architecture proposed originally for NLP [3]. Owing to
their remarkable ability to capture complex patterns and relationships in
data, transformers have been adapted for various applications, including CV
and wireless communications. This is because they have several advantages over
classical DNNs. First, transformers employ the attention mechanism, which
allows them to dynamically weigh the importance of segments in the data, which
allows them to capture short- and long-term dependencies in data. This is
particularly beneficial in time series analysis where extended sequences of
data points are involved, enhancing their capability of generalizing and
adapting to changing environments. Second, transformers take advantage of
parallel processing, which significantly improves their efficiency. Finally,
pre-trained transformer models, such as BERT [3], can be fine-tuned for
customized objectives using a small dataset (realizing transfer learning).
These advantages make transformers attractive for challenging DL tasks,
including wireless communication applications. This section delves into the
workings of transformers and examines their key components. It also examines
the architecture of the TMAE.

###  III-A Transformer Architecture

There is a variety of transformer architectures depending on the application.
However, all architectures share the same fundamental principle: the attention
mechanism. The main components of transformers, as shown in Fig. 1, are input
embedding, positional encoding, and multi-head attention, which are discussed
next.

Input Embedding: First, the input vector is segmented and projected into the
embedding space (usually of higher dimension than the input). This can be
achieved using a single convolutional layer for example. The result of this
step is a representation in the embedding space of segments of the input
vector, each of which is characterized by a position.

Figure 1:  Transformer Architecture (reproduced from [3]).

Positional Encoding: Since the transformer encoder has no recurrence (unlike
RNN), it is essential to add segment position into the data. This is
accomplished with positional encoding. There are numerous methods for
positional encoding, one of which employs trigonometric functions. In this
case, each odd indexed segment is encoded using samples of a cosine function
with a frequency that depends on the position, effectively encoding this
positional information in the generated vector. Similarly, samples of a sine
function are used to encode the positions of even indexed segments. These
positional encoding vectors are then added to the input embeddings of their
respective segments.

Multi-head Attention: Multi-head attention is the most important component of
transformers and plays a crucial role in quantifying the relationships between
the inputs. This is achieved using the self-attention mechanism, which relates
the inputs with different positions in a sequence to provide a comprehensive
representation of that sequence. Let the resulting vector obtained after
position encoding of segment $i\in\\{1,\ldots,n\\}$ be denoted by a vector
$\mathbf{x}_{i}$, and $\mathbf{X}=[\mathbf{x}_{1},\ldots,\mathbf{x}_{n}]$. As
shown in Fig. 2, three NNs are used to generate three matrices from
$\mathbf{X}$ which are the Key $\mathbf{K}$, the Query $\mathbf{Q}$, and the
Value $\mathbf{V}$, each column of which represents one segment. Then, an
attention map is generated by calculating the product
$\mathbf{Q}^{T}\mathbf{K}$, and applying a soft-max function to map the
resulting values to probabilities. The probabilities are then used as weights
multiplied by $\mathbf{V}$ to obtain a self-attended feature map for
$\mathbf{X}$. To construct multi-head attention, this attention mechanism is
applied multiple times in parallel, and the resulting outputs are concatenated
and projected again to obtain the final result. The rationale behind using
multiple attention blocks is to enable the attention function to extract
information from different perspectives (queries) and capture the complex
relationships between segments. Residual connections and layer normalization
are used to improve stability and performance, and a feed-forward NN is used
to extract more complex features.

To attend to relations between previous transformer outputs and current
inputs, a similar mechanism is applied to generate self-attended feature maps
from previous outputs, and then to generate an attention map between the
current input and previous outputs (right-most blocks in Fig. 1). Finally, the
result is converted to a representation that depends on the task at hand
(translation, prediction, classification, etc.).

Figure 2: A multi-head attention block where the scaled dot-product attention
is applied $h$ times.

Based on this architecture, the authors of [6] proposed the so-called Vision
Transformer (ViT) for CV applications. Several other transformer-based
architectures were proposed including the TMAE [7] discussed next.

Figure 3: Masked Auto-encoder Architecture: Encoding is performed on the small
subset of visible patches. The masked portions of the image are added after
the encoder, and a decoder reconstructs the original image from the complete
set of encoded patches and mask tokens [7].

###  III-B The TMAE Architecture

A TMAE is a transformer-based architecture that aims to reconstruct data using
only partial observations. Fig. 3 shows an example where the data is an image
and the transformer-based architecture is a ViT. The TMAE consists of an
encoder and a decoder.

TMAE encoder: The encoder in a TMAE is designed to convert partial
observations of the input data into a latent representation. It is realized
using a transformer-based NN applied on data with masked segments. The
transformer-based NN embeds the observed segments, adds their positional
information, and then passes the result through multiple multi-head attention
and DNN blocks as explained above (Fig. 1).

TMAE decoder: The TMAE decoder uses the latent representation and the
positions of the masked segments to reconstruct the original data as shown in
Fig. 3.

Note that the TMAE encoder is usually narrower than the TMAE decoder since the
former operates on the observed segments only whereas the latter operates on
both the observed and masked ones. Also, the TMAE encoder is usually deeper
than the TMAE decoder because it needs to learn the semantics and correlations
of the data from the observed segments. Hence, the encoder and decoder are
asymmetric.

The potential of TMAE in NG networks is highly promising, offering a novel
paradigm to address complex challenges. By amalgamating the power of
transformers with the reconstruction capabilities of an autoencoder, the TMAE
framework can efficiently process sequential data and capture intricate
temporal dependencies inherent in wireless channels. This makes it applicable
for enhancing resource efficiency, adaptability, and robustness in tasks such
as signal processing, channel estimation, and resource allocation,
contributing to the evolution of resilient and high-performance wireless
networks.

###  III-C Transformer Challenges

The superiority of transformer-based NNs comes at the cost of some challenges.
Their parallel nature increases the resources needed for them to run.
Transformers also require a substantial amount of labeled data for training,
often necessitating pre-training on large corpora before fine-tuning on the
target task. Additionally, fine-tuning pre-trained transformers on specific
tasks requires careful balancing to avoid catastrophic forgetting and retain
previously learned knowledge [8]. Note however that some of these challenges
are shared with classical DNNs. A comparison between classical DNNs and
transformer-based NNs is given in Table I, providing a concise overview of
their strengths and limitations, which helps in understanding their
suitability for different tasks.

TABLE I: Comparative Evaluation of Deep Learning Techniques Based on Key Characteristics Aspect | MLP | CNN | RNN | LSTM | Transformer
---|---|---|---|---|---
Architecture | Fully Connected | Convolutional | Recurrent | Recurrent | Attention-based
Sequence Modeling | No | Partial | Yes | Yes | Yes
Long-term Dependencies | Limited | Limited | Yes | Moderate | Improved
Parallel Processing | No | Yes | No | No | Yes
Spatial Invariance | No | Yes | No | No | No
Parameter Sharing | No | Yes | Yes | Yes | Yes
Interpretability | Low | Moderate | Low | Moderate | Moderate
Computational Complexity | Low | Moderate | Low | High | High
Memory and Storage Requirements | Low | Low | Moderate | High | High

In the next section, we provide a case study illustrating how a TMAE enhances
the compression rate of existing compression schemes.

##  IV Case Study: TMAE-Enhanced Compression

In some communication applications, communicating nodes have limited
computational resources and are deployed in resource-constrained environments.
Examples include Unmanned Aerial Vehicle (UAV) and Internet of Things (IoT)
applications. Yet, the demand for transmitting vast amounts of data (whether
it be images, sensor readings, or text) persists. Given the limited resources
in these systems, transmitting vast amounts of data becomes challenging.
Fortunately, such data often exhibits inherent correlations between its
segments. For instance, in an image, certain patches can be inferred from
their neighboring patches due to the visual structure and semantics of the
scene. Similarly, in a sequence of sensor readings, some values can be
predicted based on the preceding and succeeding values. Thus, one can omit
certain segments of the data, leveraging the correlations to infer these
segments at the receiver’s end. This approach not only reduces the amount of
data to be transmitted but also requires less processing resources at the
transmitter, making it particularly advantageous for systems like UAVs
capturing and transmitting real-time images or IoT devices sending frequent
sensor updates.

###  IV-A Example: UAV with TMAE-Enhanced Compression

Figure 4: Remote UAV imaging: A qualitative comparison between the
conventional JPEG compression and the proposed JPEG-TMAE compression scheme.

Consider a UAV operating in a distant, hard-to-reach region, tasked with
gathering essential information about the landscape through imaging. The
primary challenge for this UAV is not merely capturing the images but
efficiently transmitting them back to a base station (BS). In this scenario,
the communication channel quality is poor, limiting the amount of information
that can be sent. This requires compressing the image aggressively, which can
compromise its meaningful interpretation. Conventional compression techniques
may fail in this scenario. Fig. 4 shows an example where a standard
compression scheme produces a very low-quality image when the compression (in
bits per pixel (BPP)) is low.

TMAE-enhanced compression offers a distinct advantage. Once an image is
captured, the UAV segments it into non-overlapping patches and randomly masks
some of them, relying on the ability of the BS to infer them from unmasked
patches due to the potentially high correlation between patches. Once the
patches are masked and removed, the remaining patches are stacked to form a
condensed image, which is then further compressed using a standard compression
algorithm. Upon receipt at the BS, the compressed image is decompressed to
retrieve the stacked patches, and the TMAE, trained to handle random masks,
reconstructs the masked patches to reproduce the complete image. For accurate
reconstruction, the TMAE requires information about the location of the masked
patches. This mask can be obtained by sharing the seed of the random number
generator used to generate the mask (or can be stored at the receiver if it is
deterministic). While the resulting image might not be of the highest quality,
it conveys essential and interpretable information from the UAV despite the
communication challenges, as can be seen in Fig. 4.

Next, we examine the proposed scheme quantitatively when JPEG is used as the
standard compression algorithm.

###  IV-B JPEG-TMAE Compression

The proposed JPEG-TMAE compression scheme is a combination of the widely-used
JPEG compression and the TMAE, as shown in Fig. 4. To implement this, we use
patches of size $16\times 16$ pixels (as an example). Then, we use a masking
ratio of $R_{\rm mask}=0.67$ (i.e., $\nicefrac{{2}}{{3}}$ of the image is
masked) and random masking with a seed that is shared between the UAV and the
BS. Thus, the compression gain achieved by the TMAE is the ratio between the
size of the original and the stacked patches $(1-R_{\rm mask})$. The remaining
patches are stacked and compressed using JPEG. As such, the overall
compression rate equals the masking ratio multiplied by the JPEG compression
rate. Upon receiving the compressed image, the BS decompresses it using JPEG,
and then uses a pre-trained TMAE to reconstruct the image.

Leveraging the pre-trained transformers-based model from Facebook AI Research
(FAIR) [7], we bypassed extensive training, focusing instead on architectural
modifications tailored to our compression goals. This strategic approach
ensured an optimal balance between computational efficiency and performance in
the described UAV communication scenario.

Fig. 4 shows a comparison where JPEG and the proposed JPEG-TMAE schemes are
used to compress an image, using a compression rate of $0.41$ and $0.255$ BPP
for JPEG, and a compression rate of $0.255$ BPP for the JPEG-TMAE scheme. The
JPEG scheme fails at the compression rate $0.255$ BPP, contrary to the
proposed JPEG-TMAE which reconstructs the image with a much better quality at
the same overall compression rate (a quality comparable with JPEG at $0.41$
BPP).

For a quantitative comparison, we can use the structural similarity measure
(SSIM) to assess the quality of the reconstructed images [9]. SSIM gauges the
similarity between the input and output images, factoring in the interpixel
dependencies of closely situated pixels. Recognized as a full reference
metric, SSIM measures image quality using an uncompressed or noise-free
initial input image as its reference. Thus, we examine the SSIM of the
proposed JPEG-TMAE on the Kodak dataset [10] (often used to compare
compression methods). Through a methodical process of iterating the masking
ratio and adjusting the JPEG compression rate for every iteration, we were
able to delineate an optimized performance spectrum across all examined
masking ratios. We compare with several leading models, specifically mbt2018
(CNN-Based) [11], cheng2020-anchor (CNN-Based) [12], and ConvLSTM (CNN-LSTM
Based) [13].

Fig 5 displays the SSIM in relation to the overall compression rate. Notably,
the JPEG-TMAE not only outperforms JPEG, particularly at lower compression
rates, but it also exhibits superior performance compared to leading models
such as mbt2018, cheng2020-anchor, and ConvLSTM at these rates. In the realm
of moderate compression rates, it matches the performance of these models. An
added advantage is its design approach: while the aforementioned methods are
based on autoencoders and thus require part of the DNN architecture to be
incorporated at the UAV—increasing its complexity, our JPEG-TMAE seamlessly
integrates the entire NN onto the BS, thus simplifying processing at the UAV.
Note that while the ConvLSTM method performs best at higher compression rates,
we expect a similar transformer–based architecture to perform even better.

In summary, this approach is highly suitable for communication applications
with limited resources at the transmitter side (UAVs, IoT, etc.). Note that
the results in Fig. 5 can be further improved using dynamic masking to ensure
sufficient correlation between masked/unmasked patches, leading to either a
lower compression rate or better image quality.

$0.1$$0.15$$0.2$$0.25$$0.3$$0.35$$0.4$$0.45$$0.5$0.10.150.20.250.30.350.40.450.5Bit
per pixel (BPP)SSIMJPEGJPEG-TMAE (Optimized)ConvLSTM (CNN-LSTM
Based)[13]cheng2020-anchor(CNN based)[12]mbt2018(CNN Based)[11] Figure 5:
Comparison of the JPEG-TMAE compression scheme with conventional JPEG
compression and state-of-the-art models on the Kodak dataset.

##  V Applications in NG Networks

As demonstrated above, the TMAE can be used efficiently to reduce the size of
an image and reconstruct it using its semantics. This improves source coding
and consequently increases the throughput of wireless communications. In this
section, we discuss applications of the TMAE in various areas of wireless
communication.

###  V-A Semantic Source and Channel Coding

In Sec. IV, we present an application of the TMAE for image compression. In
general, the source coding benefits of TMAEs extend beyond image compression.
The proposed TMAE-enhanced compression can be extended to video data for
instance. However, for video data, the masks will have both space and time
dimensions. Similarly, TMAEs can be combined with standard compression schemes
as in Sec. IV to improve the compression of text or audio data. This source
coding potential can increase the throughput of NG networks.

In addition to source coding, TMAEs can be beneficial in channel coding.
Autoencoders in general have been used in the literature for various
transmission processing applications including channel coding and modulation
[2]. As an autoencoder with attention, the TMAE can be superior in such tasks.
In particular, the attention mechanism gives TMAEs an edge in joint source-
channel coding applications [14]. Also, their ability to learn long-term
dependencies can be beneficial for communications over changing communication
environments. Combining TMAEs with various NN architectures or classical
coding schemes (LDPC, polar codes) to achieve superior performance is a
promising research direction.

###  V-B Channel Estimation and Prediction

One stringent requirement in NG networks is to guarantee extremely low
latency. One way to achieve this is to shrink the packet size to reduce the
decoding transmission and computation time. However, the pilot sequences used
for channel estimation increase the packet size. In addition, in dynamic
environments where users are highly mobile, the channels change rapidly which
requires more frequent pilot signal transmission. Researchers used NNs for
channel estimation/prediction to mitigate this problem. However, the
complexity and dynamicity of wireless channels limit the capability of
existing NNs to improve channel estimation performance. We envision the TMAE
to be a strong candidate for reducing pilot signal transmission overhead due
to its superiority in reconstructing signals from partial observations.

For example, in OFDM, the channel time-frequency response can be envisioned as
a 2D image (with colors representing the real and imaginary parts) [15]. The
TMAE can be exploited for reducing the number of estimated channels, by
estimating the channels of some time-frequency resource blocks, and inferring
the rest using the TMAE by treating them as masked patches in the 2D image.
This idea can also be applied to multiple antenna systems where the channels’
spatial, temporal, and spectral characteristics need to be estimated. Another
example is channel estimation for reconfigurable intelligent surfaces (RIS),
which are surfaces that can be mounted on buildings to reflect signals in a
desired direction and improve channel quality. Despite their advantages, they
require large channel estimation overheads. A TMAE can be used to reduce this
overhead, by estimating parts of the RIS channel matrices, and exploiting the
spatial correlation in these matrices to reconstruct the rest of the channel
using a TMAE.

###  V-C Privacy and Security

Data collection is a major concern for NG networks. For applications involving
DL, collaborative learning in the cloud provides a means for processing large
amounts of data collected from distributed nodes. However, this comes at the
expense of privacy when the application involves using sensitive data or when
the trained model is shared. Intelligent NG networks should adopt privacy-by-
design approaches that are service-oriented and privacy-preserving. Several
privacy challenges can be addressed using a TMAE to build generative models to
construct privacy-preserving datasets, where data privacy and utility can be
ensured simultaneously. For instance, a TMAE can be used to encrypt user data
before it is transmitted over the network, so that it is only accessible to
authorized users who have access to the appropriate mask.

A TMAE can also be useful for detecting potential for attacks on the network
infrastructure itself. It can be used to analyze network traffic and detect
anomalies or suspicious patterns with higher accuracy than classical DNNs. In
general, by leveraging the power of the TMAE, more effective and efficient
solutions for securing networks and protecting users’ data can be developed.

##  VI Challenges and Opportunities of TMAE in NG Networks

Although the integration of TMAE into NG networks holds transformative
promise, it also brings to the forefront a series of multifaceted challenges
and open problems. These intricacies arise from the combined complexity of
transformer models and dynamic wireless networks. In this section, we discuss
the current challenges and future research directions of TMAE in NG networks.

  * •

Computational resources: Although the TMAE benefits from parallel processing
to achieve remarkable performance, this comes at the cost of computational
resources (Graphics Processing Units (GPUs)). In the case study above, we
benefited from the ability to migrate the TMAE processing to the BS which can
be equipped with a GPU. However, if the processing has to take place at a
resource-constrained device for latency considerations (such as an autonomous
vehicle), this will require this device to be equipped with expensive hardware
(computational resources and memory) which may not always be practical. The
lack of such hardware will reduce the performance or increase the latency of
TMAEs, which limits their use in real-time applications. Resolving this
limitation requires innovative techniques for distributing the TMAE between
devices and edge clouds (federated transformer-based NN). Moreover, like other
DNNs, the scalability of the TMAE for use in large-scale communication
networks presents challenges in terms of computational resources.

  * •

Energy consumption: In mobile and battery-operated wireless devices, energy
efficiency is a critical concern. The computational complexity and memory
requirements of TMAEs (and more generally transformer-based NNs) can lead to
high energy consumption, making them less energy-efficient for resource-
constrained devices. For energy-critical wireless communication networks,
researchers need to explore model compression techniques, data quantization
methods, and hardware acceleration to reduce energy consumption while
maintaining satisfactory performance.

  * •

Pre-trained models: Pre-trained transformers have shown extraordinary
performance in CV and NLP, and excellent adaptability between different
applications therein. It is expected that this property of transformers
transfers well to wireless communications applications. However, existing pre-
trained transformer models (trained on images or text) are not optimized for
wireless communications data, and their fine-tuning on such data may not
provide the best performance. On the other hand, transformer models that are
trained on wireless communications data do not exist nowadays. There is a
great opportunity for researchers to develop pre-trained transformer models
that are trained on wireless communications data (such as channels and
resource allocation data), and to demonstrate the adaptability of these models
within different applications in wireless communications with minimal training
requirements. This also highlights another opportunity, which is the
development of a large dataset of wireless communications data for such
purposes. The availability of diverse and abundant data from wireless
communications applications will enable efficient training and improved
generalization within wireless communications applications.

In summary, incorporating TMAE into future wireless communication systems
holds unparalleled potential, but addressing these intertwined challenges and
open problems is fundamental to their seamless deployment. By collectively
tackling these intricacies, the wireless communication community can usher in
a new era of adaptive, efficient, and secure communication networks.

##  VII Conclusion

In this article, we discussed the limitations of using classical deep-learning
methods in wireless networks. We then presented the architecture of
transformers and masked autoencoders and discussed their distinct capabilities
compared to traditional deep-learning methods. We also showed an application
of transformer-masked autoencoders in data compression, which yielded a
significant improvement compared to classical approaches. We explored and
presented some applications and open research problems, where transformer-
masked autoencoder-based solutions can be developed to produce intelligent
wireless communication systems. In general, it is expected that transformer-
based neural networks will play an important role in next-generation
communication networks, and there are many challenges and opportunities in
this area for the research community to explore.

## References

  * [1] Y. Xu, G. Gui, H. Gacanin, and F. Adachi, “A survey on resource allocation for 5G heterogeneous networks: Current research, future trends, and challenges,” _IEEE Commun. Surveys Tuts_ , vol. 23, no. 2, pp. 668–695, 2021\.
  * [2] Y. Sun, M. Peng, Y. Zhou, Y. Huang, and S. Mao, “Application of machine learning in wireless networks: Key techniques and open issues,” _IEEE Commun. Surveys Tuts_ , vol. 21, no. 4, pp. 3072–3108, 2019.
  * [3] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. 31st Int. Conf. Neural Info. Process. Sys._ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.
  * [4] X. Chen, Y. Hu, Z. Dong, P. Zheng, and J. Wei, “Transformer operating state monitoring system based on wireless sensor networks,” _IEEE Sensors J._ , vol. 21, no. 22, pp. 25 098–25 105, 2021.
  * [5] Z. Qin, H. Ye, G. Y. Li, and B.-H. F. Juang, “Deep learning in physical layer communications,” _IEEE Wireless Commun._ , vol. 26, no. 2, pp. 93–99, 2019\.
  * [6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _Int. Conf. Learning Representations_ , virtual, 2021.
  * [7] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _2022 IEEE/CVF Conf. Computer Vision and Pattern Recognition_ , 2022, pp. 15 979–15 988.
  * [8] M. Li, P. Ling, S. Wen, X. Chen, and F. Wen, “Bubble-wave-mitigation algorithm and transformer-based neural network demodulator for water-air optical camera communications,” _IEEE Photonics J._ , pp. 1–10, 2023.
  * [9] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE Trans. Image Process._ , vol. 13, no. 4, pp. 600–612, 2004.
  * [10] Kodak, “Kodak lossless true color image suite,” http://r0k.us/graphics/kodak/, 1999.
  * [11] D. Minnen, J. Ballé, and G. Toderici, “Joint autoregressive and hierarchical priors for learned image compression,” in _Annual Conf. Neural Inf. Processing Systems (NeurIPS), 3-8 December 2018, Montréal, Canada_ , 2018, pp. 10 794–10 803.
  * [12] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in _Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [13] G. Toderici, D. Vincent, N. Johnston, S. J. Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” _CoRR_ , vol. abs/1608.05148, 2016. [Online]. Available: http://arxiv.org/abs/1608.05148
  * [14] Y. Jiang, H. Kim, H. Asnani, S. Kannan, S. Oh, and P. Viswanath, “Joint channel coding and modulation via deep learning,” in _IEEE 21st Int. Workshop Signal Process. Advances Wireless Commun. (SPAWC)_ , Aug. 2020, pp. 1–5.
  * [15] M. Soltani, V. Pourahmadi, A. Mirzaei, and H. Sheikhzadeh, “Deep learning-based channel estimation,” _IEEE Commun. Lett._ , vol. 23, no. 4, pp. 652–655, 2019.

|  Abdullah Zayat (S’21) received B.S. degree in electrical engineering from
King Fahd University of Petroleum and Minerals (KFUPM), Saudi Arabia, in 2021.
Between 2019 and 2020, he was a Research Intern at King Abdullah University of
Science and Technology (KAUST). He then obtained his M.A.Sc. degree in
electrical engineering from the University of British Columbia (UBC), Canada,
in 2023. Currently, he is a PhD student in Electrical Engineering at UBC. His
research interests focus on signal processing, communications, and the
application of machine/deep learning in communications.
---|---
|  Mahmoud A. Hasabelnaby (S’20) received the B.S. (Hons.) and M.S. degrees in
electronics and electrical communications engineering from Menoufia
University, Menouf, Egypt, in 2014 and 2019, respectively. He is currently
working toward a Ph.D. degree in electrical engineering at the University of
British Columbia, Okanagan Campus, Kelowna, Canada. He is on leave from the
Faculty of Electronic Engineering, Menoufia University, Menouf, Egypt. From
2016 to 2018, he was a Research Assistant at the National Telecommunications
Regularity Authority, Egypt. His research interests include wireless
communications, information theory, ML/AI, end-to-end cloud-native software
development, and next-generation wireless access networks.
---|---
|  Mohanad Obeed received the B.Eng. degree in computer and communication
engineering from Taiz University, Taiz, Yemen, in 2008, the M.Sc. and the
Ph.D. degree in electrical engineering from King Fahd University of Petroleum
and Minerals (KFUPM), Dhahran, Saudi Arabia, in 2016 and 2019, respectively.
From July 2017 to July 2019, he was a visiting researcher at King Abdullah
University of Science and Technology (KAUST) under the supervision of Mohamed-
Slim Alouini. He was a Postdoctoral Research Fellow with the School of
Engineering at the University of British Columbia, Canada, from 2019 to 2022.
He joined Carleton University, as a postdoctoral research fellow, in 2023. His
research interests include satellite communication, 5G and 6G networks,
channel estimation, deep learning, and federated learning.
---|---
|  Anas Chaaban (S’09 - M’14 - SM’17) received the Maîtrise ès Sciences degree
in electronics from Lebanese University, Lebanon, in 2006, the M.Sc. degree in
communications technology and the Dr. Ing. (Ph.D.) degree in electrical
engineering and information technology from the University of Ulm and the
Ruhr-University of Bochum, Germany, in 2009 and 2013, respectively. From 2008
to 2009, he was with the Daimler AG Research Group On Machine Vision, Ulm,
Germany. He was a Research Assistant with the Emmy-Noether Research Group on
Wireless Networks, University of Ulm, Germany, from 2009 to 2011, which
relocated to the Ruhr-University of Bochum in 2011. He was a PostDoctoral
Researcher with the Ruhr-University of Bochum from 2013 to 2014, and with King
Abdullah University of Science and Technology from 2015 to 2017. He joined the
School of Engineering at the University of British Columbia as an Assistant
Professor in 2018. His research interests are in the areas of information
theory and wireless communications.
---|---