# CityBench: Evaluating the Capabilities of Large Language Model as World
Model

Jie Feng∗, Jun Zhang∗, Junbo Yan†, Xin Zhang†, Tianjian Ouyang†, Tianhui Liu†,

Yuwei Du†, Siqi Guo†, Yong Li‡

Department of Electronic Engineering, Tsinghua University, Beijing, China

###### Abstract

Large language models (LLMs) with powerful generalization ability has been
widely used in many domains. A systematic and reliable evaluation of LLMs is a
crucial step in their development and applications, especially for specific
professional fields. In the urban domain, there have been some early
explorations about the usability of LLMs, but a systematic and scalable
evaluation benchmark is still lacking. The challenge in constructing a
systematic evaluation benchmark for the urban domain lies in the diversity of
data and scenarios, as well as the complex and dynamic nature of cities. In
this paper, we propose CityBench, an interactive simulator based evaluation
platform, as the first systematic evaluation benchmark for the capability of
LLMs for urban domain. First, we build CitySim to integrate the multi-source
data and simulate fine-grained urban dynamics. Based on CitySim, we design 7
tasks in 2 categories of perception-understanding and decision-making group to
evaluate the capability of LLMs as city-scale world model for urban domain.
Due to the flexibility and ease-of-use of CitySim, our evaluation platform
CityBench can be easily extended to any city in the world. We evaluate 13
well-known LLMs including open source LLMs and commercial LLMs in 13 cities
around the world. Extensive experiments demonstrate the scalability and
effectiveness of proposed CityBench and shed lights for the future development
of LLMs in urban domain. The dataset, benchmark and source codes are openly
accessible to the research community via https://github.com/tsinghua-fib-
lab/CityBench.

11footnotetext: Equal contribution.22footnotetext: Equal
contribution.33footnotetext: Corresponding author, email:
<EMAIL_ADDRESS>
##  1 Introduction

Recent years, large language models (LLMs) with strong generalization and
reasoning capabilities have achieved excellent results in various fields [1,
42], e.g., programming [13], mathematics [48], VQA [23], common sense
reasoning [40, 32] and so on. Furthermore, powerful LLMs enable many
unimaginable research endeavors to become feasible, e.g., agent [44], embodied
intelligence [36] and AGI [9]. These researchers postulate that LLMs, by
acquiring extensive world knowledge and common sense, hold the key to
unlocking promising outcomes in these challenging applications. Many works [1,
10, 37] have demonstrated that LLMs do indeed possess a wealth of world
knowledge and can be regarded as world models to some extent, while other
works [49, 53] indicate that LLMs are not real world models and they lack an
comprehensive understanding of the real world. However, these research efforts
have primarily focused on small-scale world modelling, such as those works
conducted in virtual-home [35], while overlooking the large-scale modeling in
urban space [2, 57].

Various works have explored the potential of LLMs in modeling urban space and
solving urban tasks. For example, researchers evaluate the potential of LLMs
on remote sensing understanding tasks [19, 55] and urban visual tasks [50,
12]. Gurnee et al. [10] and Roberts et al. [37] evaluate whether LLMs acquire
the spatial knowledge of the world, such as cities and coordinates. Manvi et
al. [31, 30] try to extract the geospatial knowledge in LLMs to conduct
geospatial indicator prediction tasks [28]. Besides, researchers also explore
how to apply LLMs into the realistic urban applications, e.g., traffic control
[5, 20], traffic prediction [22], mobility prediction [46], visual language
navigation [38] and so on. However, on the one hand, these existing works
primarily focus on evaluating the static spatial knowledge of LLMs without
considering the environment dynamics and interactivity, which are the key
point for the world model [21]. On the other hand, most of them only focus on
one type of task and one modality of data in the urban space, using small
dataset that are not scalable globally. The root cause of their shortcomings
is the insufficient integration of multi-source data for urban spaces and the
lack of suitable interactive city simulators. Although there are some existing
simulators for urban space such as game simulators [14] and traffic simulators
[26], they are either not open-source, or tailored to specific problems. All
of them do not support systematically evaluating the capability of LLMs as
world models.

Figure 1: The framework of global evaluation benchmark CityBench, which
consists of a simulator CitySim and 7 representative urban tasks. We can
select any city around the world to automatically build new benchmark for it.

In this paper, we propose CityBench, a comprehensive evaluation platform for
assessing the capability of LLMs as city-scale world models. It covers
multiple modalities, supports interactive simulations, and is easily
extensible globally. CityBench consists of two modules: a simulation module
CitySim for integrating multi-source urban data and simulating urban dynamics,
a evaluation module Benchmark for various evaluation of LLMs. In CitySim, we
first collect three kinds of open source urban data: geospaital data from Open
Street Map, urban vision data including from Google Map, and human activity
data from Foursquare and other websites. Then, we build an efficient GPU based
engine to simulate individual behaviors in the urban environment and develop
various interfaces for controlling the urban dynamics and sensing the urban
environments. Furthermore, based on CitySim, we design a comprehensive
benchmark to evaluate the capability of LLMs as city-scale world models,
covering core research problems from various urban research fields. The
evaluation benchmark comprises two levels of tasks: geospatial understanding
tasks and decision-making tasks. In geospatial-understanding tasks, based on
the integrated multi-source data from CitySim, we introduce street
view&satellite image understanding and urban space knowledge understanding
tasks to evaluate the basic capability of LLMs as city-scale world models. In
decision-making tasks, we apply LLMs to interact with CitySim to complete the
mobility prediction task, traffic signal control task and street navigation
task which require comprehensive ability of LLMs as city-scale world models.
In summary, our contribution are as follows,

  * •

We build CitySim, an efficient simulator for integrating multi-source urban
data and simulating find-grained individual behaviors in the urban
environments. It provides ease-of-use APIs for controlling urban dynamics and
sensing urban environments.

  * •

We propose CityBench, a comprehensive evaluation benchmark for evaluating the
capability of LLMs as city-scale world models, which includes 4 multi-modal
geospatial understanding tasks and 3 interactive urban decision-making tasks.

  * •

Extensive experiments on CityBench with 13 well-known open source and
commercial LLMs demonstrate the effectiveness of CityBench as evaluation
benchmark and also present the limitation of applying LLMs as world models in
urban tasks.

##  2 Related Work

LLM Evaluation for Urban Space Researchers from various urban related fields
have conduct extensive evaluations of LLM in urban space from different
aspects. Zhang et al [55] and Kuckreja et al. [19] evaluate the performance of
multi-modal LLMs on several remote sensing related tasks. Yang et al. [52]
propose V-IRL benchmark to evaluate the performance of multi-modal LLMs on
street view image related tasks including localization and recognition tasks.
Mai et al. [28] and Manvi et al. [31] use LLMs to predict social indicators
like population and education level. Gurnee et al. [10] and Bhandari et al.
[3] try to testify whether LLMs know the coordinates of geospatial entity.
Mooney et al. [33] and Deng et al. [6] use GIS exams to understand the
gospatial skills of LLMs. Roberts et al. [37] design GPT4GEO to evaluate the
geospatial capacity of LLM with limited case studies. Different from these
works with single aspect, our work covers most of the above problems in one
benchmark with different modalities and views. Furthermore, we also support
interactive simulation based various decision making task evaluation which
enable more realistic and challenging performance evaluation.

Interactive Decision Making and Urban Simulator Beyond the above static
evaluation, researchers also evaluate the capacity of LLMs in the interactive
decision making tasks with customized simulators, e.g., web agent [24, 54]
with web environment and embodied intelligence [53] with virtual home [35]. In
the urban domain, Schumann et al. [38] apply LLM to do the visual language
navigation task in Touchdown [4] and Lai [20] apply LLMs as the traffic light
controller in CityFlow [56] to manage the road traffic. Besides, Yang et al.
[52] design V-IRL as the environment of street view image related tasks and
propose a global scale virtual intelligence benchmark. These works only
evaluate the potential of LLMs in single urban decision making task and most
of their results rely on small-scale datasets in limited regions. Different
from them, our work builds on an efficient urban simulator with global scale
and supports three representative urban decision making tasks with different
modality in one benchmark, including mobility prediction [46], traffic control
[5] and street navigation. In the future, it will support more kinds of urban
tasks, e.g., urban planning [58].

##  3 CitySim

In the section, we present an overview of the simulation module CitySim. The
module first integrates multi-source urban data to support multi-modal urban
sensing and urban dynamic simulation. Next, in order to reconstruct real urban
dynamics and individual behaviors, the module also provides a efficient GPU-
based simulation engine with ease-to-use sensing and controlling APIs.

###  3.1 Datasets

To present a complete picture of the city’s geospatial structure, semantic
features, and human activities, CitySim integrates the following globally
available data from multiple sources.

Geospatial Data Geospatial data, represented by maps, is the most fundamental
data for describing the urban structure including buildings, road networks,
green areas, etc. OpenStreetMap (OSM) 111https://openstreetmap.org/ is an open
source map data provider that provides researchers with globally available
geospatial data. However, the raw data provided by OSM cannot support the
simulation of urban dynamics directly, and the relationship between different
elements is incomplete such as the connection between buildings and roads.
Therefore, we provide a globally available map building tool
222https://github.com/tsinghua-fib-lab/mosstool that reconstructs lanes, lane
topology, and building-lane connections based on the raw OSM data. The
reconstructed map is used as the geospatial base and simulation input in
CitySim. Based on the map, APIs such as search by ID, search by relationship,
and search by spatial range are provided in the CitySim module to help LLMs
access the city geospatial information.

Urban Visual Data Street view data and satellite images are two types of urban
data that contains rich semantic information, which can be used to evaluate
the capacity of multi-model LLMs. In addition, the two types of data are also
publicly available globally. Therefore, CitySim also integrates the two types
of data, the former obtained via Google Maps API and Baidu Maps API, and the
latter using the Esri World
Imagery333https://www.arcgis.com/home/item.html?id=10df2279f9684e4a9f6a7f08febac2a9
as data source. In the CitySim module, street view data is accessed through
spatial location and facing direction, and satellite images are acquired
through spatial ranges.

Human Activities Data We use the open-source global Foursquare-checkin [51]
data and a synthetic global origin-destination data (OD data) [25] as the
proxy of human activities to enable the fine-grained human movement
simulation. The Foursquare-checkin [51] dataset is a long-term user checkin
data collected from Foursquare 444https://foursquare.com/ around 415 cities in
the world. Origin-destination data is generated by a diffusion model with
population from Worldpop555https://www.worldpop.org/ and satellite image from
Esri World Imagery as input. While all the user information are anonymized, we
follow the license from Foursquare-checkin [51] to protect the public privacy.

###  3.2 Simulation

Cities have strong dynamics triggered by human activities, such as traffic
flow. Such dynamics are a strong challenge to the capacity of LLMs. To
simulate urban dynamics, we implement GPU-accelerated individual-level
simulations of human activities. The simulator takes the map reconstructed
from OSM and the travel demand described by the synthetic OD data, and
simulates the mobility behaviors through individual motion models such as the
car-following model [43] and the lane-change model [18] to obtain the
information about the dynamics of all pedestrians and vehicles in the city
every second. The simulator also provides a series of sensing and control
APIs. Through the sensing APIs, LLM can obtain data about urban dynamics such
as pedestrian location, vehicle speed, and road average speed. Through the
control API, LLM can intervene in the city’s operation, such as modifying
traffic signal lights, modifying the speed limit of the road, etc.

Overall, with the CitySim module, we provide globally available integrated
multi-source data with an interactive urban dynamic simulator to multi-model
LLMs.

##  4 CityBench

Based on CitySim, we introduce a global multi-modal urban evaluation benchmark
CityBench to evaluate the capability of multi-modal LLMs as the city-scale
world models with 7 urban tasks. These tasks are 4 perception and
understanding tasks and 3 decision making tasks, detailed design of these
tasks are introduced as follows.

###  4.1 Perception and Understanding

In this section, we introduce the perception and understanding task group. The
first task is the street view image geolocalization task from the urban visual
intelligence [8]. Following are social indicator prediction and infrastructure
inference tasks from remote sensing field. Finally, we adapt GeoQA [29] task
into urban environment.

####  4.1.1 Image Geolocalization

Image geolocalization task is to predict the precise location of image based
its context. Street view image is regarded as the recording of urban
appearance and play an important role in understanding the urban environment
and dynamics [8]. Thus, we query multi-modal LLM with street view image and
require it to directly generate the city name of its location and precise
latitude and longitude coordinates. A good multi-modal LLM should recognize
the important objects from the street image and mapping them into the
potential locations. Following [11], we define two metrics: city inference
accuracy and accuracy within 1km/25km of the correct location to measure the
performance.

####  4.1.2 Geospatial Prediction

Geospatial predictions are important for understanding the global sustainable
development especially for developing countries, e.g., poverty estimation [15]
and population density estimation [41]. One of the most widely used solutions
is using satellite images with machine learning methods to predict these
socioeconomic indicators. In the benchmark, following the setting from [31],
we query multi-modal LLM with a satellite image as context to predict the
population density of it. We use population dataset from Worldpop [41] as the
groundtruth and Pearson’s $r^{2}$ and RMSE as metrics.

####  4.1.3 Infrastructure Inference

Besides, we also introduce the infrastructure inference task which means to
recognize the urban infrastructures from the satellite images. This task
require the ability of scene understand and object segmentation of urban
environment. The groundtruth of urban infrastructure of each satellite image
is extracted from the OSM by matching predefined infrastructure key words
within a fixed spatial range. Given the satellite image and the list of all
kinds of infrastructures, multi-modal LLM is required to generate the
infrastructure names appeared in the image. In the benchmark, we pay attention
to the following infrastructures: Airport, Harbor, Stadium, Bridge, Roundabout
and Train Station.

####  4.1.4 GeoQA for City Elements

Beyond understanding the urban space from the visual perspective, we introduce
geographic question answer(GeoQA) [29] to test whether LLMs comprehends the
fundamental spatial elements [27] in a city from the concept view, such as
road, landmarks and boundary. For example, we directly ask LLM about the
relation between different roads in a city. Following [27], we classify the
spatial elements into six groups and design several single choice problems for
each group. These six groups are node, path, landmark, boundary, districts and
others.

###  4.2 Planning and Decision Making

Different from the static evaluation introduced in the last section, we design
three interactive decision making tasks to evaluate the capability of LLMs in
dynamic and partial observed environments which are more challenging and
realistic. With the interaction with the CitySim simulator and dynamic human
activities, LLM needs to understand the important mechanisms and regularity in
the urban environments to complete the decision marking tasks. Detailed design
of decision making tasks are introduced as follows.

####  4.2.1 Mobility Prediction

Mobility prediction task is one of the fundamental task for understanding the
human behaviors and urban dynamics. Mobility prediction task is to predict the
next location of user in the next time window with given the past mobility
trajectory. Here we use the the global foursquare checkin data as the proxy of
human mobility and integrate it into the simulator to support the mobility
prediction task. We follow [46] to conduct the mobility prediction task via
LLMs.

####  4.2.2 Street Navigation

Street navigation task is widely used in neurocognitive science [7] and visual
language navigation [38] as the important benchmark for evaluating the spatial
cognition of human and models. Here, we also define a text based street
navigation task to evaluate the zero-shot navigation capability of LLMs in a
new city. Different from the visual language navigation which require model to
follow the language instruction and understand the scene via street view
image, our street navigation task require model to explore the region via the
local information (e.g. accessed road names) provided by the simulator during
action and its intrinsic knowledge of the whole urban environment.

####  4.2.3 Traffic Signal Control

Traffic signal control task is one of the widely studied realistic urban
decision making task in recent years [47]. It is challenging for existing
methods due to the dynamic traffics and the generalization issues. Traffic
signal control task is to generate the future traffic signal schedule by
considering the current traffic states and the future traffics. Lai et al.
[20] propose LLMLight to solve the single traffic signal control in simple
intersection and demonstate the generalization of LLMs. Follow this work, we
extend it into multiple traffic signal control in a region with more than one
intersections, which are more challenging and require cooperation between
different intersections.

##  5 Benchmark and Experiments

###  5.1 Settings

Dataset We collect multi-source data for 13 cities around the world, including
street view& satellite image, geospatial data and human activity data.
Detailed information about these dataset are presented in Table 1. Our
platform can expand to any city in the world with just inputting its name. The
effortless evaluation across various regions and cultures makes it a effective
world model evaluator of LLMs. CityBench complies with the Google Maps
Platform license666https://cloud.google.com/maps-platform/terms, the Baidu
Maps Platform license777https://map.baidu.com/zt/client/service/index.html and
the Esri Master License Agreement888https://www.esri.com/en-
us/legal/terms/full-master-agreement.

Table 1: Multi-source data for 13 cities around the world. For simplifying the
processing steps, we have defined these spatial range of collected data within
rectangular boundaries. Most cities only include the central part of the
city’s administrative boundaries, which may lead to some inaccuracies.
Specific coordinate settings can be found in the appendix.

World | Cities | Visual Data | GeoSpatial Data | Human Activity Data
---|---|---|---|---
| Satellite
---
Image
| Street View
---
(sampling)
Roads | PoI/AoIs |  | OD flow
---
(>10)
Checkins
Asia | Beijing | 1764 | 1000 | 17043 | 276090 | 1905025 | 21015
Shanghai | 5925 | 1000 | 33321 | 57731 | 845188 | 33129
Mumbai | 638 | 1000 | 6296 | 60245 | 309147 | 31521
Tokyo | 1120 | 1000 | 33174 | 1146094 | 969865 | 1044809
Europe | London | 1710 | 1000 | 14418 | 83892 | 1401404 | 173268
Paris | 238 | 1000 | 4443 | 21950 | 28362 | 85679
Moscow | 1558 | 1000 | 9850 | 28289 | 979064 | 836313
Americas | NewYork | 320 | 1000 | 5414 | 349348 | 71705 | 390934
SanFrancisco | 345 | 1000 | 4171 | 73777 | 61367 | 100249
SaoPaulo | 1332 | 1000 | 28714 | 1681735 |  311830 | 808754
Africa | Nairobi | 336 | 1000 | 2972 |  264101 | 135332 | 25727
CapeTown | 896 | 1000 | 5947 | 151711 | 525578 | 11591
Oceania | Sydney | 1935 | 1000 | 21390 | 141997 | 438763 | 54170

Baselines We select state-of-the-art open source LLMs and commercial API to
evaluate the performance of LLMs on the benchmark. For multi-modal LLMs, open
source LLMs are LLaVa-NeXT [23], CogVLM-v2 [45], MiniCPM-LLama3-V-2.5 [34],
commercial APIs are Qwen-VL-plus and GPT4o. For LLMs, open source LLMs are
LLama3-8B, LLama3-70B, Mistral-7B-v0.2 [16], Mixtral-8x22B-v0.1 [17] and
DeepSeekv2 [39], commercial APIS are GPT3.5, GPT4 [1]. We test most of the
open-source LLMs by using the API from DeepInfra 999https://deepinfra.com/,
while the commercial models using their official APIs
101010https://platform.openai.com/docs/api-reference/introduction. We also
deploy several multi-modal LLMs with 1 A100 GPU, including LLaVa-NeXT [23],
CogVLM-v2 [45], MiniCPM-LLama3-V-2.5 [34].

Evaluation Metrics We follow the common practice of each task to define the
metrics. Metrics and instances for each task are presented in Table 2. For
each task with results from 13 cities, we report the mean value and standard
deviation value of them in the following section.

Table 2: Detailed information of 7 evaluation tasks in CityBench, including
data modality, metric and data instances. Task settings across different
cities keep consistent.

CityBench | Tasks | Modality | Metrics | Instances
---|---|---|---|---
Perception& Understanding | Image Geolocalization | Image | Accuracy, Accuracy@1km/25km | 500 $\times$ 13 cities
Geospatial Prediction | Image |  $r^{2}$, RMSE | ~500 $\times$ 13 cities
Infrastructure Inference | Image | Accuracy, Recall | ~500 $\times$ 13 cities
GeoQA for City | Text | Accuracy | ~1000 $\times$ 13 cities
Planning& Decision Making | Mobility Prediction | Text | Top1-Acc, F1 | 500 $\times$ 13 cities
Street Navigation | Text | Steps, Success Rate | 50 $\times$ 13 cities
Traffic Signal Control | Text | Travel Time, Queue Length, Throughput | 1 regions $\times$ 3600s $\times$ 13 cities

###  5.2 Main Results

####  5.2.1 Perception and Understanding

Results of visual perception tasks of multi-modal LLMs are presented in Table
3. In general, commercial LLMs like GPT4o performs best in most tasks,
achieving the best results in 5 out of 7 metrics. Among the open source
models, LLaVA-NeXT-34B performs best with achieving 3 second best results in 7
metrics. Besides, we can find that the performance of open source LLM varies
significantly across different cities which indicates significant geospatial
bias of open source LLMs. Detailed analysis about this can refer to section
5.2.3. Results of geospatial knowledge QA task of LLMs are presented in Table
4. As Table 4 shows, commercial LLMs GPT4-Turbo performs best in 4 out of 6
tasks and the left two performs second best. In summary, in terms of
geospatial knowledge about urban spaces, the open-source LLMs lag far behind
the commercial LLMs.

Table 3: Results of multi-modal LLMs on three representative visual
understanding tasks. For most metrics, higher values are better, except for
RMSE. In the table, bold denotes the best results, underline denotes the
second best results.

Models | Image Geolocalization | Population Density | Infrastructures
---|---|---|---
| City
---
Accuracy
| Accuracy
---
@1km
| Accuracy
---
@25km
$r^{2}$ | RMSE | Precision | Accuracy
Open Source | LLaVA-NeXT-8B | 0.267$\pm$0.322 | 0.0097$\pm$0.0269 | 0.221$\pm$0.337 | -0.764$\pm$0.582 | 3.31$\pm$0.823 | 0.245$\pm$0.301 | 0.796$\pm$0.187
LLaVA-NeXT-34B | 0.501$\pm$0.387 | 0.0123$\pm$0.0328 | 0.408$\pm$0.413 | -0.163$\pm$0.603 | 2.61$\pm$0.643 | 0.262$\pm$0.297 | 0.804$\pm$0.187
CogVLM2 | 0.559$\pm$0.333 | 0.0067$\pm$0.0176 | 0.326$\pm$0.335 | -0.301$\pm$0.737 | 2.75$\pm$0.504 | 0.287$\pm$0.320 | 0.726$\pm$0.262
MiniCPM-V2.5 | 0.262$\pm$0.267 | 0.0061$\pm$0.0124 | 0.223$\pm$0.270 | -1.054$\pm$0.740 | 3.57$\pm$0.621 | 0.251$\pm$0.323 | 0.806$\pm$0.197
Commercial API | Qwen-VL-plus | 0.793$\pm$0.288 | 0.0052$\pm$0.0039 | 0.645$\pm$0.313 | -1.028$\pm$1.667 | 3.14$\pm$1.016 | 0.203$\pm$0.233 | 0.454$\pm$0.170
GPT4o | 0.862$\pm$0.079 | 0.0187$\pm$0.0375 | 0.797$\pm$0.148 | 0.122$\pm$0.309 | 2.32$\pm$0.664 | 0.248$\pm$0.306 | 0.812$\pm$0.195

Table 4: Results of different LLMs on GeoQA for city, a geospatial knowledge
question answering task for urban space. Here, the higher value is better,
bold denotes the best results, underline denotes the second best results.

| Models | Nodes | Landmarks | Paths | Districts | Boundary | Others
---|---|---|---|---|---|---|---
Open Source | LLama3-8B | 0.2322$\pm$0.0150 | 0.2765$\pm$0.0143 | 0.3173$\pm$0.0054 | 0.2627$\pm$0.0163 | 0.3569$\pm$0.0109 | 0.3365$\pm$0.0225
Mistral-7B | 0.2286$\pm$0.0167 | 0.1950$\pm$0.0083 | 0.2054$\pm$0.0179 | 0.1511$\pm$0.0150 | 0.3269$\pm$0.0232 | 0.2688$\pm$0.0186
LLama3-70B | 0.2646$\pm$0.0076 | 0.3304$\pm$0.0320 | 0.4048$\pm$0.0334 | 0.2532$\pm$0.0429 | 0.3727$\pm$0.0112 | 0.3485$\pm$0.0185
Mixtral-8x22B | 0.2633$\pm$0.0155 | 0.3569$\pm$0.0367 | 0.3652$\pm$0.0201 | 0.2730$\pm$0.0370 | 0.3415$\pm$0.0125 | 0.3239$\pm$0.0241
Commercial API | DeepSeekV2 | 0.2904$\pm$0.0065 | 0.3035$\pm$0.0165 | 0.4133$\pm$0.0208 | 0.3518$\pm$0.0399 | 0.4650$\pm$0.0363 | 0.3264$\pm$0.0232
GPT3.5-Turbo | 0.2477$\pm$0.0260 | 0.2627$\pm$0.0128 | 0.3222$\pm$0.0255 | 0.1909$\pm$0.0244 | 0.3788$\pm$0.0137 | 0.3050$\pm$0.0244
GPT4-Turbo | 0.3429$\pm$0.0095 | 0.3938$\pm$0.0393 | 0.5249$\pm$0.0349 | 0.3054$\pm$0.0412 | 0.4485$\pm$0.0198 | 0.3707$\pm$0.0219

####  5.2.2 Panning and Decision Making

Table 5 presents the results of mobility prediction task and street navigation
task, the performance difference between LLMs in decision making tasks are
much smaller than the last section. It is amazing to observe that open source
LLMs (e.g., LLama3-70B) performs better than commercial LLMs in these two
tasks. Results of traffic signal control task are presented in Table 6. Here,
we report results from three cities with various traffic volume and
environment. In general, commercial LLMs like GPT-3.5 or GPT-4 performs better
but small open source LLMs can also achieve competitive results in some cases.
One interesting direction is to investigate these difference and find out the
underlying factors behind them.

Table 5: Results of LLMs on mobility prediction and street navigation tasks.
The arrow next to the metric indicates the direction of improvement, an upward
arrow indicates that higher values of the metric are better.

Models | Mobility Prediction | Street Navigation
---|---|---
| Top1
---
Accuracy $\uparrow$
F1 $\uparrow$ |  | Success
---
Rate $\uparrow$
| Average
---
Steps $\downarrow$
Open Source | LLama3-8B | 0.130$\pm$0.0320 | 0.094$\pm$0.030 | 0.747$\pm$0.140 | 5.304$\pm$2.059
Mistral-7B | 0.090$\pm$0.0255 | 0.087$\pm$0.031 | 0.730$\pm$0.134 | 5.382$\pm$1.982
LLama3-70B | 0.159$\pm$0.0430 | 0.130$\pm$0.034 | 0.796$\pm$0.085 | 4.941$\pm$1.592
Mixtral-8x22B | 0.155$\pm$0.0419 | 0.136$\pm$0.042 | 0.745$\pm$0.133 | 5.339$\pm$1.961
Commercial API | DeepSeekV2 | 0.126$\pm$0.0520 | 0.101$\pm$0.044 | 0.698$\pm$0.119 | 5.739$\pm$1.812
GPT3.5-Turbo | 0.152$\pm$0.0388 | 0.113$\pm$0.035 | 0.719$\pm$0.143 | 5.473$\pm$2.015
GPT4-Turbo | 0.147$\pm$0.0357 | 0.125$\pm$0.035 | 0.757$\pm$0.091 | 5.184$\pm$1.520

Table 6: Results of LLMs on traffic signal tasks on three cities with
representative traffic volume. We used real-world mobility demands to generate
traffic to support the simulation. The traffic signals we actually control in
the experiment are only a part of the trajectory path, while the remaining
signals are operated according to the default fixed-time scheduler.

Models | Paris | Newyork | Beijing
---|---|---|---
| Throughput
---
| Travel
---
Time
| Queue
---
Length
| Throughput
---
| Travel
---
Time
| Queue
---
Length
| Throughput
---
| Travel
---
Time
| Queue
---
Length
LLama3-8B | 205 | 2936.579 | 652.157 | 580 | 2635.365 | 86.993 | 2128 | 1873.757 | 40.941
Mistral-7B | 183 | 2940.694 | 651.413 | 574 | 2634.054 | 87.807 | 1696 | 2059.84 | 57.759
Llama3-70B | 193 | 2937.528 | 650.855 | 516 | 2664.102 | 89.116 | 2031 | 1893.517 | 43.475
Mixtral-8x22B | 179 | 2938.643 | 653.168 | 526 | 2655.111 | 88.567 | 1604 | 2093.986 | 63.006
DeepSeekv2 | 193 | 2938.567 | 653.532 | 495 | 2682.194 | 89.914 | 2233 | 1754.923 | 32.771
GPT3.5-Turbo | 214 | 2935.621 | 651.704 | 544 | 2649.214 | 87.926 | 2199 | 1778.963 | 34.965
GPT4-Turbo | 191 | 2937.580 | 652.026 | 495 | 2670.238 | 89.706 | 2240 | 1746.608 | 32.234

####  5.2.3 Geospatial Bias Analysis

To further investigate the difference between LLMs, we report the detailed
results of mobility prediction task and image geolocalization tasks from 13
cities in Figure 2. Based on the above results, we have made several
interesting discoveries. First, we find that the performance of different LLMs
varies a lot across different cities, no LLM can always perform best in
mobility prediction tasks. Second, we find that the performance of multi-modal
LLMs on visual task like image geolocalization task are significantly biased.
Most LLMs perform well in major international cities, but poorly in some
lesser-known cities (e.g., CapeTown). The variability in evaluation results
demonstrate the necessity of establishing a global evaluation benchmark, and
also highlights the potential shortcomings and areas for improvement of LLMs.

Figure 2: Detailed results of LLMs on mobility prediction task(left) for
different cities and image geolocalization task(right) for different models.
In left figure, the order of cities in each group is decided by the prediction
performance of GPT3.5: Tokyo, Paris, London, Mumbai, Moscow, CapeTown,
Nairobi, SanFrancisco, Sao Paulo, Shanghai, Sydney, NewYork, Beijing.

##  6 Discussion

Limitations. While our platform is based on the public data from various
sources, the quality of different data may play a important role in the
evaluation results. For example, The varying levels of map data and street
view data quality across different regions worldwide may significantly
negatively impact the results. Besides, due to the limitation of global label
data for various task, we only select 7 urban tasks in current benchmark which
only cover portion of the research problems in urban domain. In the future, we
plan to collect more kinds of tasks with global scale groundtruth data to
further improve the reliability and representativeness of benchmark.

Ethical considerations and potential societal impact. Our benchmark is
designed for enable the global evaluation of LLMs for various cities with
different cultures and countries. We try our best to improve the ease-of-use
and fairness for cities with different development levels. However, as
discussed in section 5.2.3, due to the limitation of accessed data, the
evaluation results for different cites varies a lot. Therefore, the variation
in evaluation results caused by data quality may lead to a certain degree of
misunderstanding regarding the performance on some urban problems. We call the
whole community for attention to this issue to improve the usability of LLMs
across different races and countries, promoting fairness and sustainable
development of the world.

Develop foundation model for urban domain. Based on the results of our
benchmark, we find existing LLMs perform poorly on many urban tasks, even
worse than some classic simple baseline algorithms. Developing LLMs tailored
for urban domain is urgently necessary. We hope our benchmark can accelerate
this development and we look forward to a more comprehensive and robust
evaluation framework for urban domain.

##  7 Conclusion

In this paper, we propose CityBench, a systematic evaluation benchmark for
LLMs in urban domain. In CityBench, we first build CitySim to integrate multi-
source urban data and provide the interactive ability of simulating fine-
grained urban dynamics. Based on CitySim, we design 7 important urban tasks to
constitute the CityBench for evaluating the capabilities of LLMs in the urban
domain from various aspects. Results on 13 cities around the world demonstrate
the effectiveness and scalability of CityBench. In the future, we plan to
further expand CityBench to include a wider range of tasks, thereby
facilitating more robust and representative evaluation of LLMs for urban
domain research.

## References

  * [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] M. Batty, K. W. Axhausen, F. Giannotti, A. Pozdnoukhov, A. Bazzani, M. Wachowicz, G. Ouzounis, and Y. Portugali.  Smart cities of the future.  The European Physical Journal Special Topics, 214:481–518, 2012.
  * [3] P. Bhandari, A. Anastasopoulos, and D. Pfoser.  Are large language models geospatially knowledgeable?  In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, pages 1–4, 2023.
  * [4] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi.  Touchdown: Natural language navigation and spatial reasoning in visual street environments.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12538–12547, 2019.
  * [5] L. Da, M. Gao, H. Mei, and H. Wei.  Llm powered sim-to-real transfer for traffic signal control.  arXiv preprint arXiv:2308.14284, 2023.
  * [6] C. Deng, T. Zhang, Z. He, Q. Chen, Y. Shi, L. Zhou, L. Fu, W. Zhang, X. Wang, C. Zhou, et al.  Learning a foundation language model for geoscience knowledge understanding and utilization.  arXiv preprint arXiv:2306.05064, 2023.
  * [7] R. A. Epstein, E. Z. Patai, J. B. Julian, and H. J. Spiers.  The cognitive map in humans: spatial navigation and beyond.  Nature neuroscience, 20(11):1504–1513, 2017.
  * [8] Z. Fan, F. Zhang, B. P. Loo, and C. Ratti.  Urban visual intelligence: Uncovering hidden city profiles with street view images.  Proceedings of the National Academy of Sciences, 120(27):e2220417120, 2023.
  * [9] T. Feng, C. Jin, J. Liu, K. Zhu, H. Tu, Z. Cheng, G. Lin, and J. You.  How far are we from agi.  arXiv preprint arXiv:2405.10313, 2024.
  * [10] W. Gurnee and M. Tegmark.  Language models represent space and time.  arXiv preprint arXiv:2310.02207, 2023.
  * [11] L. Haas, S. Alberti, and M. Skreta.  Pigeon: Predicting image geolocations.  arXiv preprint arXiv:2307.05845, 2023.
  * [12] X. Hao, W. Chen, Y. Yan, S. Zhong, K. Wang, Q. Wen, and Y. Liang.  Urbanvlp: A multi-granularity vision-language pre-trained foundation model for urban indicator prediction.  arXiv preprint arXiv:2403.16831, 2024.
  * [13] S. Hong, X. Zheng, J. Chen, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, et al.  Metagpt: Meta programming for multi-agent collaborative framework.  arXiv preprint arXiv:2308.00352, 2023.
  * [14] P. Interactive.  "cities: Skylines ii", 2023.
  * [15] N. Jean, M. Burke, M. Xie, W. M. Davis, D. B. Lobell, and S. Ermon.  Combining satellite imagery and machine learning to predict poverty.  Science, 353(6301):790–794, 2016.
  * [16] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al.  Mistral 7b.  arXiv preprint arXiv:2310.06825, 2023.
  * [17] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al.  Mixtral of experts.  arXiv preprint arXiv:2401.04088, 2024.
  * [18] A. Kesting, M. Treiber, and D. Helbing.  General lane-changing model mobil for car-following models.  Transportation Research Record, 1999(1):86–94, 2007.
  * [19] K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan.  Geochat: Grounded large vision-language model for remote sensing.  arXiv preprint arXiv:2311.15826, 2023.
  * [20] S. Lai, Z. Xu, W. Zhang, H. Liu, and H. Xiong.  Large language models as traffic signal control agents: Capacity and opportunity.  arXiv preprint arXiv:2312.16044, 2023.
  * [21] Y. LeCun.  A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27.  Open Review, 62(1), 2022.
  * [22] C. Liu, S. Yang, Q. Xu, Z. Li, C. Long, Z. Li, and R. Zhao.  Spatial-temporal large language model for traffic prediction.  arXiv preprint arXiv:2401.10134, 2024.
  * [23] H. Liu, C. Li, Q. Wu, and Y. J. Lee.  Visual instruction tuning.  Advances in neural information processing systems, 36, 2024.
  * [24] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al.  Agentbench: Evaluating llms as agents.  arXiv preprint arXiv:2308.03688, 2023.
  * [25] loooffeeeey.  Generate origin-destination matrix based-on public available information on the internet, 2024.
  * [26] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner.  Microscopic traffic simulation using sumo.  In The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018.
  * [27] K. Lynch.  The image of the city.  MIT press, 1964.
  * [28] G. Mai, W. Huang, J. Sun, S. Song, D. Mishra, N. Liu, S. Gao, T. Liu, G. Cong, Y. Hu, et al.  On the opportunities and challenges of foundation models for geospatial artificial intelligence.  arXiv preprint arXiv:2304.06798, 2023.
  * [29] G. Mai, K. Janowicz, R. Zhu, L. Cai, and N. Lao.  Geographic question answering: challenges, uniqueness, classification, and future directions.  AGILE: GIScience series, 2:8, 2021.
  * [30] R. Manvi, S. Khanna, M. Burke, D. Lobell, and S. Ermon.  Large language models are geographically biased.  arXiv preprint arXiv:2402.02680, 2024.
  * [31] R. Manvi, S. Khanna, G. Mai, M. Burke, D. Lobell, and S. Ermon.  Geollm: Extracting geospatial knowledge from large language models.  arXiv preprint arXiv:2310.06213, 2023.
  * [32] G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom.  Gaia: a benchmark for general ai assistants.  arXiv preprint arXiv:2311.12983, 2023.
  * [33] P. Mooney, W. Cui, B. Guan, and L. Juhász.  Towards understanding the geospatial skills of chatgpt: Taking a geographic information systems (gis) exam.  In Proceedings of the 6th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, pages 85–94, 2023.
  * [34] OpenBMB.  Minicpm-llama3-v 2.5, 2024.
  * [35] X. Puig, K. Ra, M. Boben, J. Li, T. Wang, S. Fidler, and A. Torralba.  Virtualhome: Simulating household activities via programs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018.
  * [36] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, et al.  A generalist agent.  arXiv preprint arXiv:2205.06175, 2022.
  * [37] J. Roberts, T. Lüddecke, S. Das, K. Han, and S. Albanie.  Gpt4geo: How a language model sees the world’s geography.  arXiv preprint arXiv:2306.00020, 2023.
  * [38] R. Schumann, W. Zhu, W. Feng, T.-J. Fu, S. Riezler, and W. Y. Wang.  Velma: Verbalization embodiment of llm agents for vision and language navigation in street view.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18924–18933, 2024.
  * [39] Z. Shao, D. Dai, D. Guo, and B. Liu.  Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.  2024\.
  * [40] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al.  Challenging big-bench tasks and whether chain-of-thought can solve them.  arXiv preprint arXiv:2210.09261, 2022.
  * [41] A. J. Tatem.  Worldpop, open data for spatial demography.  Scientific data, 4(1):1–4, 2017.
  * [42] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [43] M. Treiber, A. Hennecke, and D. Helbing.  Congested traffic states in empirical observations and microscopic simulations.  Physical review E, 62(2):1805, 2000.
  * [44] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al.  A survey on large language model based autonomous agents.  Frontiers of Computer Science, 18(6):1–26, 2024.
  * [45] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, J. Xu, B. Xu, J. Li, Y. Dong, M. Ding, and J. Tang.  Cogvlm: Visual expert for pretrained language models, 2023.
  * [46] X. Wang, M. Fang, Z. Zeng, and T. Cheng.  Where would i go next? large language models as human mobility predictors.  arXiv preprint arXiv:2308.15197, 2023.
  * [47] H. Wei, G. Zheng, V. Gayah, and Z. Li.  A survey on traffic signal control methods.  arXiv preprint arXiv:1904.08117, 2019.
  * [48] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * [49] J. Xiang, T. Tao, Y. Gu, T. Shu, Z. Wang, Z. Yang, and Z. Hu.  Language models meet world models: Embodied experiences enhance language models.  Advances in neural information processing systems, 36, 2024.
  * [50] Y. Yan, H. Wen, S. Zhong, W. Chen, H. Chen, Q. Wen, R. Zimmermann, and Y. Liang.  Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web.  In The Web Conference 2024, 2024.
  * [51] D. Yang, D. Zhang, and B. Qu.  Participatory cultural mapping based on collective behavior data in location-based social networks.  ACM Transactions on Intelligent Systems and Technology (TIST), 7(3):1–23, 2016.
  * [52] J. Yang, R. Ding, E. Brown, X. Qi, and S. Xie.  V-irl: Grounding virtual intelligence in real life.  arXiv preprint arXiv:2402.03310, 2024.
  * [53] M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel.  Learning interactive real-world simulators.  arXiv preprint arXiv:2310.06114, 2023.
  * [54] A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang.  Agenttuning: Enabling generalized agent abilities for llms.  arXiv preprint arXiv:2310.12823, 2023.
  * [55] C. Zhang and S. Wang.  Good at captioning, bad at counting: Benchmarking gpt-4v on earth observation data.  arXiv preprint arXiv: 2401.17600, 2024.
  * [56] H. Zhang, S. Feng, C. Liu, Y. Ding, Y. Zhu, Z. Zhou, W. Zhang, Y. Yu, H. Jin, and Z. Li.  Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario.  In The world wide web conference, pages 3620–3624, 2019.
  * [57] Y. Zheng, L. Capra, O. Wolfson, and H. Yang.  Urban computing: concepts, methodologies, and applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 5(3):1–55, 2014.
  * [58] Z. Zhou, Y. Lin, D. Jin, and Y. Li.  Large language model for participatory urban planning.  arXiv preprint arXiv:2402.17161, 2024.

##  Appendix A Appendix

###  A.1 Parameter Settings

In the experiments, the default generation parameters for LLMs are presented
in Table 7. Besides, we modify some parameters for some experiments. For
street navigation tasks, the temperature is changed to 0.7 for better
exploration. For multi-modal LLM, the temperature of CogVLM2 is 0.8 and the
temperature of MiniCPM-V-2.5 is 0.7.

Table 7: Generation parameter settings for LLMs during the experiments.

Pameters |  | Open Source
---
LLM(DeepInfra)
| Commercial
---
API
| Open Source
---
MLLM
| Commercial API
---
for MLLM
temperature | 0 | 0 | 0.2 | 1.0
top p | 1.0 | 1.0 | 1.0 | 1.0(GPT4o), 0.8(Qwen-VL-plus)
max tokens | 512 | 512 | 512 | 100
presence penalty | 0 | 0 | 0 | 0
frequency penalty | 0 | 0 | 0 | 0
repetition penalty | 1 | 1 | 1 | 1

###  A.2 Spatial range of 13 Cities

Selected spatial range of 13 cities are presented in Figure 3.

Figure 3: Maps for Beijing, Shanghai, Cap Town, London, Moscow, Mumbai,
Nairobi, San Francisco, Sao Paulo, Sydney, Tokyo.


#  cellanneal: A user-friendly deconvolution software for omics data

Lisa Buchauer and Shalev Itzkovitz
Department of Molecular Cell Biology, Weizmann Institute of Science, Rehovot,
Israel

###### Abstract

We introduce cellanneal, a python-based software for deconvolving bulk RNA
sequencing data. cellanneal relies on the optimization of Spearman’s rank
correlation coefficient between experimental and computational mixture gene
expression vectors using simulated annealing. cellanneal can be used as a
python package or via a command line interface, but importantly also provides
a simple graphical user interface which is distributed as a single executable
file for user convenience. The python package is available at
https://github.com/LiBuchauer/cellanneal, the graphical software can be
downloaded at http://shalevlab.weizmann.ac.il/resources.

_K_ eywords mixture deconvolution $\cdot$ bulk deconvolution $\cdot$ graphical
user interface $\cdot$ python package $\cdot$ command-line interface $\cdot$
single-cell methods

##  1 Introduction

Single-cell sequencing methods enable precise characterization of gene
expression patterns in individual cells. However, they may provide inaccurate
information about the cell type composition of samples, as required
preprocessing procedures such as tissue dissociation or cell sorting affect
viability of different cell types to varying extent [1]. Further, especially
in the clinical context, single-cell sequencing of patient samples is
currently not routinely applied because of high cost and required expertise,
while bulk sequencing is more prevalent.

For these reasons, computational deconvolution methods are gaining popularity
in basic and clinical research. Computational deconvolution approaches infer
the cell type proportions constituting a given bulk RNA sample based on
separately obtained cell type reference data. Several computational
deconvolution methods have been developed in the last decade and have
contributed to our understanding of tissue composition [2, 3]. Generally,
during deconvolution, the computational mixture is constructed from a set of
cell type fractions and reference gene expression vectors for each of the
participating cell types, most commonly derived from single-cell data. The
cell type fractions are then iteratively changed until agreement between the
_in silico_ gene expression vector and the observed bulk sample gene
expression vector is optimal by a measure of choice. Here, published methods
rely almost exclusively on minimizing the sum of squared residuals between
bulk and computationally mixed vectors. Algorithms for such optimization
problems are readily available and include variants of least squares
regression (e.g. weighted least squares regression [4], non-negative least
squares regression [5, 6] or least trimmed squares [7]) and support vector
regression [8, 9].

However, least squares-based optimization is faced with a particular challenge
in bulk RNAseq deconvolution because of the highly skewed nature of mRNA copy
number distributions, ranging from less than 1 to more than 10,000 average
mRNA copies per cell [10, 11]. In such settings, optimization results may be
strongly influenced by few highly expressed genes and are thus not robust to
noise or platform effects influencing the readout of these genes. Identifying
the right genes for deconvolution becomes a task in itself [12]. As a result,
deconvolution methods may yield inferred mixed gene expression vectors that do
not correlate well with measured bulk gene expression.

##  2 Statement of Need

Making sense of bulk RNA sequencing datasets often requires analysis of the
cell type composition of the samples. This is particularly relevant in
clinical samples that analyze the transcriptome of tissues or tumors which
consist of epithelial, stromal and immune cell types. In parallel, publicly
available single-cell data sets enable precise characterization of the
expression signature of multiple individual cell types. However, software
tools for computational bulk deconvolution are often slow, non-robust and not
easy to use. Some existing methods address the aspect of user-friendliness by
providing graphical web interfaces, but submitting sensitive medical data to
an external web server is not always compatible with privacy legislation.

To address these challenges, we have developed cellanneal, a deconvolution
approach that uses Spearman’s rank correlation coefficient between synthetic
and bulk gene expression vectors as the optimization procedure’s objective
function. Because this correlation measure is calculated from ranks rather
than absolute data values, each gene influences the optimization result to a
similar extent. Users are encouraged to include as many informative genes as
possible in the analysis. cellanneal optimizes cell type fractions by
simulated annealing, a flexible, rapid and robust algorithm for global
optimization [13, 14]. cellanneal can be used as a python package, via its
command line interface or via a user-friendly graphical software which runs
locally. Its typical processing time for one mixture sample is below one
minute on a desktop machine.

##  3 Availability and Features

The python package and command line interface are available at
https://github.com/LiBuchauer/cellanneal and can be installed using pip. The
graphical software for Microsoft Windows and MacOS can be downloaded at
http://shalevlab.weizmann.ac.il/resources and does not require installation.
Instructions for installation and use as well as general documentation is
available at https://github.com/LiBuchauer/cellanneal.

The python package provides functions for the three main steps of a
deconvolution analysis with cellanneal: identification of a gene set for
deconvolution, deconvolution using simulated annealing, and plotting the
results. A quick start workflow is available as part of the documentation. For
the command line interface and the graphical user interface, these three steps
are combined into one call (click).

cellanneal runs which were started from either the command line or the
graphical user interface produce a collection of result files including
tabular deconvolution results (cell type fractions for each sample) and
figures illustrating these cell type distributions. Further, cellanneal
computes and stores the gene-wise fold change between the observed bulk
expression and the estimated expression based on the inferred cell type
composition. This enables identifying genes for which expression may be
specifically induced or inhibited in the bulk sample compared to the single
cell reference. Such genes may be of biological or medical interest.

cellanneal relies on the python packages scipy [14], numpy [15], pandas [16],
seaborn [17], matplotlib [18] and tkinter [19].

##  4 Acknowledgements

We thank all members of the Itzkovitz lab for valuable feedback and for
testing the software.

L.B. is supported by the European Molecular Biology Organization under EMBO
Long-Term Fellowship ALTF 724-2019. S.I. is supported by the Wolfson Family
Charitable Trust, the Edmond de Rothschild Foundations, the Fannie Sherr Fund,
the Helen and Martin Kimmel Institute for Stem Cell Research grant, the
Minerva Stiftung grant, the Israel Science Foundation grant No. 1486/16, the
Broad Institute-Israel Science Foundation grant No. 2615/18, the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation program grant No. 768956, the Chan Zuckerberg Initiative grant No.
CZF2019-002434, the Bert L. and N. Kuggie Vallee Foundation and the Howard
Hughes Medical Institute (HHMI) international research scholar award.

## References

  * [1] Dan D Erdmann-Pham, Jonathan Fischer, Justin Hong, and Yun S Song.  Likelihood-based deconvolution of bulk gene expression data using single-cell references.  Genome Research, 31(10):1794–1806, 2021.
  * [2] Francisco Avila Cobos, José Alquicira-Hernandez, Joseph E Powell, Pieter Mestdagh, and Katleen De Preter.  Benchmarking of cell type deconvolution pipelines for transcriptomics data.  Nature communications, 11(1):1–14, 2020.
  * [3] Gregor Sturm, Francesca Finotello, Florent Petitprez, Jitao David Zhang, Jan Baumbach, Wolf H Fridman, Markus List, and Tatsiana Aneichyk.  Comprehensive evaluation of transcriptome-based cell-type quantification methods for immuno-oncology.  Bioinformatics, 35(14):i436–i445, 07 2019.
  * [4] Julien Racle, Kaat de Jonge, Petra Baumgaertner, Daniel E Speiser, and David Gfeller.  Simultaneous enumeration of cancer and immune cell types from bulk tumor gene expression data.  elife, 6:e26476, 2017.
  * [5] Xuran Wang, Jihwan Park, Katalin Susztak, Nancy R Zhang, and Mingyao Li.  Bulk tissue cell type deconvolution with multi-subject single-cell expression reference.  Nature communications, 10(1):1–9, 2019.
  * [6] Brandon Jew, Marcus Alvarez, Elior Rahmani, Zong Miao, Arthur Ko, Kristina M Garske, Jae Hoon Sul, Kirsi H Pietiläinen, Päivi Pajukanta, and Eran Halperin.  Accurate estimation of cell composition in bulk expression through robust integration of single-cell information.  Nature communications, 11(1):1–11, 2020.
  * [7] Yuning Hao, Ming Yan, Blake R Heath, Yu L Lei, and Yuying Xie.  Fast and robust deconvolution of tumor infiltrating lymphocyte from expression profiles using least trimmed squares.  PLoS computational biology, 15(5):e1006976, 2019.
  * [8] Aaron M Newman, Chih Long Liu, Michael R Green, Andrew J Gentles, Weiguo Feng, Yue Xu, Chuong D Hoang, Maximilian Diehn, and Ash A Alizadeh.  Robust enumeration of cell subsets from tissue expression profiles.  Nature methods, 12(5):453–457, 2015.
  * [9] Aaron M Newman, Chloé B Steen, Chih Long Liu, Andrew J Gentles, Aadel A Chaudhuri, Florian Scherer, Michael S Khodadoust, Mohammad S Esfahani, Bogdan A Luca, David Steiner, et al.  Determining cell type abundance and expression from bulk tissues with digital cytometry.  Nature biotechnology, 37(7):773–782, 2019.
  * [10] Bo Li, Eric Severson, Jean-Christophe Pignon, Haoquan Zhao, Taiwen Li, Jesse Novak, Peng Jiang, Hui Shen, Jon C Aster, Scott Rodig, et al.  Comprehensive analyses of tumor immunity: implications for cancer immunotherapy.  Genome biology, 17(1):1–16, 2016.
  * [11] Björn Schwanhäusser, Dorothea Busse, Na Li, Gunnar Dittmar, Johannes Schuchhardt, Jana Wolf, Wei Chen, and Matthias Selbach.  Global quantification of mammalian gene expression control.  Nature, 473(7347):337–342, 2011.
  * [12] Hananeh Aliee and Fabian J Theis.  Autogenes: Automatic gene selection using multi-objective optimization for rna-seq deconvolution.  Cell Systems, 2021.
  * [13] Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi.  Optimization by simulated annealing.  science, 220(4598):671–680, 1983.
  * [14] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17:261–272, 2020.
  * [15] Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  Nature, 585(7825):357–362, September 2020.
  * [16] The pandas development team.  pandas-dev/pandas: Pandas, February 2020.
  * [17] Michael L. Waskom.  seaborn: statistical data visualization.  Journal of Open Source Software, 6(60):3021, 2021.
  * [18] J. D. Hunter.  Matplotlib: A 2d graphics environment.  Computing in Science & Engineering, 9(3):90–95, 2007.
  * [19] Fredrik Lundh.  An introduction to tkinter.  URL: www. pythonware. com/library/tkinter/introduction/index. htm, 1999.