# Knowledge-infused Contrastive Learning for Urban Imagery-based Socioeconomic
Prediction

Yu Liu  Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>,  Xin Zhang
Tsinghua UniversityBeijingChina<EMAIL_ADDRESS>,  Jingtao Ding  Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>,  Yanxin Xi  University of
HelsinkiFinlandChina<EMAIL_ADDRESS>and  Yong Li  Tsinghua
UniversityBeijingChina<EMAIL_ADDRESS>
###### Abstract.

Monitoring sustainable development goals requires accurate and timely
socioeconomic statistics, while ubiquitous and frequently-updated urban
imagery in web like satellite/street view images has emerged as an important
source for socioeconomic prediction. Especially, recent studies turn to self-
supervised contrastive learning with manually designed similarity metrics for
urban imagery representation learning and further socioeconomic prediction,
which however suffers from effectiveness and robustness issues. To address
such issues, in this paper, we propose a Knowledge-infused Contrastive
Learning (KnowCL) model for urban imagery-based socioeconomic prediction.
Specifically, we firstly introduce knowledge graph (KG) to effectively model
the urban knowledge in spatiality, mobility, etc., and then build neural
network based encoders to learn representations of an urban image in
associated semantic and visual spaces, respectively. Finally, we design a
cross-modality based contrastive learning framework with a novel image-KG
contrastive loss, which maximizes the mutual information between semantic and
visual representations for knowledge infusion. Extensive experiments of
applying the learnt visual representations for socioeconomic prediction on
three datasets demonstrate the superior performance of KnowCL with over 30%
improvements on $R^{2}$ compared with baselines. Especially, our proposed
KnowCL model can apply to both satellite and street imagery with both
effectiveness and transferability achieved, which provides insights into urban
imagery-based socioeconomic prediction.

Urban knowledge graph, socioeconomic prediction, urban imagery, contrastive
learning

††ccs: Computing methodologies Knowledge representation and reasoning††ccs:
Computing methodologies Computer vision representations††ccs: Applied
computing Law, social and behavioral sciences

##  1\. Introduction

Driven by the rapid urbanization, more than half of the world population-4.4
billion inhabitants-live in cities and contribute over 80% of global GDP today
(Bank, 2022), which makes cities an increasingly important role in achieving
United Nations Sustainable Development Goals (SDGs) on economy, education,
environment, health, etc. (of Economic and Affairs, 2018, 2022). Especially,
socioeconomic indicators like population, educational background and household
income are good proxies for SDG monitoring (Gebru et al., 2017). The
traditional door-to-door surveys for such statistics however are costly,
labor-intensive, time-consuming and further affected by recent COVID-19
pandemic (of Economic and Affairs, 2022). In contrast, the inclusive,
ubiquitous and frequently-updated web applications paves the way for high-
quality, economical and timely SDG monitoring. Recently, researchers predict
socioeconomic indicators with the enormous amount of web data especially the
urban imagery (Burke et al., 2021; Li et al., 2022; Wang et al., 2018), i.e.,
the satellite imagery and the street view imagery provided in web map services
like Google Map and web platforms like Instagram.

Figure 1. Illustration of infusing various types of knowledge for urban
imagery-based socioeconomic prediction, e.g., “nearBy”, “flowTransition” and
“similarFunction” relational links describe urban knowledge in spatiality,
mobility and function (POI category distribution), respectively.

Built upon the great success of deep learning in computer vision (Deng et al.,
2009; Krizhevsky et al., 2017), most studies adopt convolutional neural
networks (CNNs) to learn visual representations of urban imagery for
socioeconomic prediction. Specifically, earlier studies follow the task-
specific supervised learning for visual representations with neighborhood
demographics (Gebru et al., 2017; Abitbol and Karsai, 2020) and country
poverty (Jean et al., 2016; Yeh et al., 2020) as supervision signals, which
require massive labeled data for training and suffer from generalization
issues (Rolf et al., 2021). To overcome such issues, recent studies turn to
self-supervised learning with contrastive objectives, a.k.a, contrastive
learning (Liu et al., 2021c), and learn a single representation vector for one
image to generalize across diverse prediction tasks (Wang et al., 2022; Ahn et
al., 2020). Based on manually designed similarity metrics, these studies learn
visual representations of urban imagery by maximizing the agreement between
similar images in latent space under an image view-based contrastive learning
framework (Zhang et al., 2020). For example, a commonly used metric on
spatiality knowledge is from Tobler’s First Law of Geography (Miller, 2004)
that spatially near images should have similar semantics and thus closer
representations (Jean et al., 2019; Wang et al., 2020a; Kang et al., 2020;
Bjorck et al., 2021). Moreover, a recent study (Xi et al., 2022) applies the
typical contrastive learning framework, SimCLR (Chen et al., 2020), for
satellite imagery-based socioeconomic prediction, assuming that images with
similar point of interest (POI) features should have closer representations
111Here POI features correspond to POI category distributions of regions
identified in urban imagery.. Owing to the task-agnostic representation
learning from unlabeled data, contrastive learning becomes a promising avenue
for urban imagery-based socioeconomic prediction.

Despite this, existing contrastive learning based methods heavily rely on
manually designed similarity metrics for urban imagery representation
learning, which only capture one or two types of semantic knowledge in urban
environment and thus affect the practical performance in socioeconomic
prediction. According to recent works of leveraging multi-source urban data
for socioeconomic prediction (Zhang et al., 2021; Wu et al., 2022), there are
various types of semantic knowledge available for urban imagery-based
socioeconomic prediction, e.g., the spatiality knowledge of spatial
neighborhood, the mobility knowledge of significant flow transitions and the
function knowledge of similar POI category distributions, as shown in Figure
1. Thus, how to infuse comprehensive knowledge into contrastive learning for
urban imagery-based socioeconomic prediction becomes an important research
problem, which however is challenging in:

  * •

Effective structure for knowledge identification. Unlike the well-known domain
knowledge like Tobler’s First Law of Geography, other types of aforementioned
knowledge lack explicit domain definition. Moreover, further knowledge
infusion requires an effective structure to store and represent the knowledge,
increasing the difficulty of knowledge identification for urban imagery-based
socioeconomic prediction.

  * •

Contrastive learning for knowledge infusion. Existing studies adopt the image
view-based contrastive learning framework where the similarity metric is
manually designed with a single type of knowledge. Therefore, they fail to
infuse various types of knowledge for urban imagery-based socioeconomic
prediction.

To address such challenges, in this paper, we propose a Knowledge-infused
Contrastive Learning model for urban imagery-based socioeconomic prediction,
termed as KnowCL. Firstly, motivated by the recent success of the structured
knowledge graph (KG) for urban knowledge modeling (Zhuang et al., 2017; Liu et
al., 2022; Liu et al., 2021b; Wang et al., 2021, 2020b), we introduce urban
knowledge graph (UrbanKG) to identify comprehensive knowledge in multi-source
urban data. In the UrbanKG, entity nodes characterize urban elements like
regions, POIs and business centers, while relation edges describe semantic
connections between them, i.e., the knowledge in spatiality, mobility and
function. Moreover, we present cross-modality based contrastive learning for
knowledge infusion by exploiting the naturally associated pairing of urban
imagery and regions in UrbanKG222The knowledge graph is identified as another
kind of modality data versus urban imagery data in visual modality.. To be
specific, for an urban image, KnowCL develops a visual encoder to extract its
visual representation, and a semantic encoder to extract KG embedding of its
associated region entity in UrbanKG (Wang et al., 2017), which are further
optimized for maximum agreement with contrastive loss on image-KG pairs. The
learnt visual representations of urban imagery are further fed into
traditional regression models for diverse socioeconomic prediction tasks.
Therefore, KnowCL firstly leverages UrbanKG for knowledge identification, then
represents comprehensive knowledge with KG embedding, and combines with cross-
modality based contrastive learning to achieve knowledge infusion for urban
imagery-based socioeconomic prediction. The main contributions of this paper
are summarized as follows:

  * •

To the best of our knowledge, we are the first to investigate KG for urban
imagery-based socioeconomic prediction, which provides an effective structure
to comprehensively identify the semantic knowledge in spatiality, mobility,
function, etc.

  * •

We propose a cross-modality based contrastive learning framework, which
infuses semantic knowledge into visual representations of urban imagery via
the novel contrastive objective between the image and KG modalities. The
proposed framework might shed light on urban imagery representation learning.

  * •

We conduct extensive experiments on three cities of Beijing, Shanghai and New
York across six socioeconomic indicators. The results on both satellite and
street view imagery demonstrate that our proposed framework achieves
significant performance improvement compared with state-of-the-art models.
Further ablation studies and analysis confirm the effectiveness and
transferability of knowledge infusion for urban imagery-based socioeconomic
prediction.

##  2\. Related Work

As described before, the urban imagery-based socioeconomic prediction studies
focus on urban imagery representation learning, which extracts visual
representations for downstream socioeconomic prediction tasks. Based on
whether the urban imagery representation learning process needs supervision
signals from downstream tasks, related studies can be classified into
supervised learning, unsupervised learning and self-supervised learning333The
self-supervised learning is separated from the unsupervised one, which
emphasizes using supervision signals generated from data itself..

Supervised Urban Imagery Representation Learning for Socioeconomic Prediction.
We first discuss about the input source of satellite imagery. With the CNN
model pre-trained on ImageNet (Deng et al., 2009) and light intensity as
supervision signal, both Jean _et al._ (Jean et al., 2016) and Yeh _et al._
(Yeh et al., 2020) extract satellite imagery representations for assets
prediction in Africa. Similar frameworks are proposed in (He et al., 2018;
Park et al., 2022) for economic indicator prediction. Han _et al._ (Han et
al., 2020a) train a teacher-student network with limited labels to predict
demographics like household and income. As for street imagery case, Gerbu _et
al._ (Gebru et al., 2017) train a CNN model to identify the types and number
of cars in street view imagery, which are further used to estimate
socioeconomic indicators like race and education. Lee _et al._ (Lee et al.,
2021) leverage semantic segmentation and graph convolution network (GCN) to
predict livelihood indicators of wealth index and BMI. Moreover, Law _et al._
(Law et al., 2019) extract features from both satellite and street view
imagery to estimate the house prices. However, above studies learn urban
imagery representations supervised by a specific downstream task, i.e., the
learnt representations cannot generalize to various socioeconomic prediction
tasks.

Unsupervised Urban Imagery Representation Learning for Socioeconomic
Prediction. Han _et al._ (Han et al., 2020b) adopt clustering algorithm and
partial order graph to distinguish economic development of satellite imagery,
which are further used to train a scoring model for urbanization prediction.
Suel _et al._ (Suel et al., 2019) apply the pre-trained CNN model to extract
street view imagery representations for inequality measurement in urban
environment. Besides, He _et al._ (He et al., 2018) extract traditional image
features like histogram of oriented gradients from both satellite and street
view imagery to predict commercial activeness. However, such unsupervised
learning methods only capture shallow features of urban imagery, which lead to
inferior performance.

Self-supervised Urban Imagery Representation Learning for Socioeconomic
Prediction. Motivated by the milestones of self-supervised learning achieved
in computer vision (Wang et al., 2022; Chen et al., 2020; Jing and Tian,
2020), researchers also leverage self-supervised learning especially
contrastive learning for urban imagery-based socioeconomic prediction, and
focus on similarity metric design to distill expressive representations of
urban imagery. Especially, most studies follow the Tobler’s First Law of
Geography (Miller, 2004) that “everything is related to everything else, but
near things are more related than distant things”, and design corresponding
similarity metrics or loss forms. For example, Jean _et al._ (Jean et al.,
2019) employs the triplet loss to minimize the distance between
representations of spatially near satellite images but maximize the distance
between those of spatially distant pairs, while Wang _et al._ (Wang et al.,
2020a) employs the similar loss for street view imagery case. Moreover, recent
studies (Kang et al., 2020; Bjorck et al., 2021; Xi et al., 2022) mainly adopt
the InfoNCE loss (Oord et al., 2018) with SimCLR framework (Chen et al., 2020)
to encode such spatiality knowledge into visual representations. Xi _et al._
(Xi et al., 2022) further incorporate a POI-based similarity metric such that
images corresponding to similar POI category distributions should have closer
visual representations. Furthermore, Li _et al._ (Li et al., 2022) consider
the spatiality based similarity metrics for both satellite imagery and street
imagery. According to the discussion above, most existing studies adopt the
image view-based contrastive learning framework, where a pair of images are
compared to capture spatiality knowledge, failing to infuse various types of
knowledge together. In comparison, our proposed KnowCL model captures
comprehensive knowledge via UrbanKG and achieves effective knowledge infusion
with cross-modality based contrastive learning framework.

##  3\. Preliminaries & Problem Statement

As stated in _The sustainable Development Goals Report_ (of Economic and
Affairs, 2022), the SDGs determine the survival of humanity, facing the
current confluence of crises. Especially, various socioeconomic indicators are
characterized for SDG monitoring (Corsi et al., 2012):

###### Definition 0 (Socioeconomic Indicator).

Socioeconomic indicators measure the status of the region/nation on the
socioeconomic scale, determined by a combination of social and economic
factors such as population, amount and kind of education, household
consumption, crime rate, etc.

Moreover, the urban region becomes an important subject for socioeconomic
indicator investigation, which are defined as:

###### Definition 0 (Urban Region).

A city can be partitioned into a set of urban regions $\mathcal{A}$, following
certain partition criteria like road network division and administrative
division (Wang et al., 2020a; Han et al., 2020a).

The urban imagery includes the satellite imagery and the street view imagery,
which are visual appearances of the city from overhead view and ground-level
view, respectively (Wang et al., 2018; Li et al., 2022). Figure 1 provides
some examples of the urban imagery. Specifically, the satellite images are
collected by satellites, which capture the structure of regions in the city.
The street view images are taken by automobiles or citizens along the street,
which capture the internal environment of regions in the city. Moreover, an
urban region usually associates with one satellite image but multiple street
view images taken at different locations therein, which are defined as:

###### Definition 0 (Urban Imagery).

Given a city, the urban imagery set is denoted by
$\mathcal{I}=\mathcal{I}^{\text{SI}}/\mathcal{I}^{\text{SV}}$ with the
satellite imagery set $\mathcal{I}^{\text{SI}}$ and the street view imagery
set $\mathcal{I}^{\text{SV}}$. For $\forall a\in\mathcal{A}$, its associated
satellite image is denoted by $I^{\text{SI}}_{a}\in\mathcal{I}^{\text{SI}}$,
and its associated $n$ street view images are denoted by
$I^{\text{SV}}_{a}=\\{I^{\text{SV}}_{a,1},\\!\cdots\\!,I^{\text{SV}}_{a,n}\\}$
with
$I^{\text{SV}}_{a,1},\\!\cdots\\!,I^{\text{SV}}_{a,n}\in\mathcal{I}^{\text{SV}}$.

The UrbanKG generalizes the commonly used KG concept (Wang et al., 2017; Hogan
et al., 2021) to urban domain for urban knowledge modeling (Zhuang et al.,
2017; Liu et al., 2022; Liu et al., 2021b; Wang et al., 2021), which is
defined as:

###### Definition 0 (Urban Knowledge Graph).

An UrbanKG is defined as a multi-relational graph
$\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{F})$, where $\mathcal{E}$,
$\mathcal{R}$ and $\mathcal{F}$ are the sets of entities, relations and facts,
respectively, with
$\mathcal{F}=\\{(e_{h},r,e_{t})|e_{h},e_{t}\in\mathcal{E},r\in\mathcal{R}\\}$
hold. Especially, the entity set $\mathcal{E}$ includes urban elements like
regions, POIs, business centers and categories, while the relation set
$\mathcal{R}$ describes their semantic connections on spatiality, mobility,
function and business. The set of region entities in $\mathcal{G}$ corresponds
to above defined region set $\mathcal{A}$.

Details of the UrbanKG will be presented in detail in Section 4.3. To capture
the semantic knowledge for downstream applications, recent studies learn to
embed entities and relations of a KG into low-dimensional vector space,
a.k.a., KG embeddings (Wang et al., 2017; Hogan et al., 2021; Vashishth et
al., 2019).

Based on the preliminaries above, we then formally define the urban imagery-
based socioeconomic prediction problem as follows.

###### Problem 1 (Urban Imagery-based Socioeconomic Prediction).

Given the region set $\mathcal{A}$ with its associated urban imagery set
$\mathcal{I}$, for $\forall a\in\mathcal{A}$, the main goal is to learn the
visual representation $\bm{I}_{a}$ and well estimate the socioeconomic
indicator $y_{a}$. The ground truth values of $y_{a}$ is assumed to be unknown
in urban imagery representation learning.

##  4\. Methodology

###  4.1. Framework Overview

Figure 2 presents the main framework of our proposed KnowCL model for urban
imagery-based socioeconomic prediction problem. Since we consider the
socioeconomic indicators on region level, we solve the challenges of knowledge
identification and knowledge infusion with focus on regions in the city.

Figure 2. The main framework of KnowCL model, where the urban imagery input
can be either satellite imagery or street view imagery. The projector heads
after encoders are omitted for simplicity in the illustration.

To identify the comprehensive knowledge in urban environment, we firstly
introduce the recently proposed UrbanKG structure to store and represent urban
knowledge related with regions, which is then fed into a GCN-based semantic
encoder to learn KG embeddings for region entities therein. As for
satellite/street view images associated with regions, a CNN-based visual
encoder is adopted for visual representations. Furthermore, we propose cross-
modality based contrastive learning framework to achieve knowledge infusion.
Especially, the designed image-KG contrastive loss encourages one region’s KG
embedding and its associated urban imagery representation to exhibit high
mutual information, through which the semantic knowledge preserved in KG
embedding is successfully infused into urban imagery representation. Finally,
the knowledge-infused urban imagery representations are leveraged for diverse
socioeconomic prediction tasks.

###  4.2. Urban Knowledge Identification

As defined in Section 3, we introduce UrbanKG to identify the urban knowledge
for socioeconomic prediction. Specifically, the entities in UrbanKG include
regions partitioned by road network, business centers of commercial and
consumption activities, POIs of infrastructures like restaurants, markets and
schools, as well as categories of POI attributes, e.g., food, shopping,
education, etc. Thus, the entity types in UrbanKG are Region, Business Center
(BC), POI and Category.

Table 1. The captured knowledge and corresponding relational structures in UrbanKG. Knowledge | Relation | Head Entity | Tail Entity
---|---|---|---
Spatiality | _borderBy_ | Region | Region
_nearBy_ | Region | Region
_locateAt_ | POI | Region
Mobility | _flowTransition_ | Region | Region
Function | _similarFunction_ | Region | Region
_coCheckin_ | POI | POI
_cateOf_ | POI | Category
Business | _provideService_ | BC | Region
_belongTo_ | POI | BC
_competitive_ | POI | POI

Moreover, we model the comprehensive knowledge in multi-source urban data as
semantic relations, which are summarized in Table 1. Various types of
knowledge are represented in triple form with relation, head entity and tail
entity.

  * •

Spatiality. We determine _borderBy_ and _nearBy_ relational links by spatial
distance between regions, and use _locateAt_ to identify POIs’ spatially
located regions.

  * •

Mobility. we aggregate individual mobility trajectories to induce the
significant flow transition between regions, which are linked by
_flowTransition_.

  * •

Function. We consider widely used features in urban computing tasks (Zheng et
al., 2014) as function knowledge. For example, _coCheckin_ connects highly
correlated POIs in terms of concurrence in check-in data, which implies two
POIs are consecutively visited by several people, e.g., a cinema and a
neighboring restaurant (Liu et al., 2021a). Since the region function is
usually featured by POI category distribution therein (Xi et al., 2022), we
connect regions with similar POI category distribution via _similarFunction_.
Besides, _cateOf_ describes the category attribute of POIs.

  * •

Business. We connect regions with their neighboring business centers via
_provideService_ , to capture the economic status of regions. Similarly, POIs
are connected with neighboring business centers via _belongTo_. To further
identify the competitiveness between POIs, neighboring POIs with the same
category are connected via _competitive_ (Li et al., 2020).

Besides, reverse edges are added to model inverse relations, e.g., (POI,
_locateAt_ , Region) and (Region, _$\sim$ locateAt_, POI). Following the
structure above, we construct the UrbanKG with urban knowledge identified for
socioeconomic prediction. The construction details can be referred to Section
A.

###  4.3. Encoder Design

####  4.3.1. Semantic Encoder Design

The semantic encoder aims to learn region embeddings with urban knowledge
represented. To fully exploit both semantic information of various relations
and structural information of graph topology in UrbanKG, we adopt the GCN-
based encoder for region embeddings (Vashishth et al., 2019; Liu et al.,
2021a).

Given the UrbanKG $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{F})$, for
$\forall v\in\mathcal{E},r\in\mathcal{R}$, their $d$-dimensional embeddings
after $l$ layer are denoted by $\bm{e}^{l+1}_{v}$ and $\bm{r}^{l+1}$,
respectively. The neighborhood of $v$ is denoted by
$\mathcal{N}_{v}=\\{(u,r)|(u,r,v)\in\mathcal{F}\\}$, and
$\bm{e}^{l+1}_{v}\in\mathbb{R}^{d}$ can be calculated as:

(1)
$\displaystyle\bm{e}^{l+1}_{v}=\sigma\left(\sum_{(u,r)\in\mathcal{N}_{v}}\bm{W}^{l}_{\text{dir}(r)}\phi(\bm{e}^{l}_{u},\bm{r}^{l})+\bm{W}^{l}_{\text{self}}\bm{e}^{l}_{v}\right),$

where $\bm{W}^{l}_{\text{dir(r)}}$ and $\bm{W}^{l}_{\text{self}}$ are
direction-specific projection matrices for incoming/outgoing relations and
self loop relation, respectively.
$\phi:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ is the
composition function for message calculation in GCN, e.g., element-wise
summation and element-wise product (Vashishth et al., 2019). $\sigma(\cdot)$
is an activation function. Besides, we use pre-trained embeddings from TuckER
(Balažević et al., 2019) for initialized embeddings.

Let $f^{\text{KG}}(\cdot)$ denote the semantic encoder with $L$ layers of GCN
following above design, and the region embedding for $a\in\mathcal{A}$ can be
calculated as $\bm{e}_{a}=f^{\text{KG}}(\mathcal{G},a)$, i.e.,
$\bm{e}_{a}=\bm{e}^{L}_{a}$.

####  4.3.2. Visual Encoder Design

Our proposed KnowCL model allows various choices of network architectures for
visual encoder design. For simplicity, we adopt the commonly used ResNet (He
et al., 2016) to obtain visual representations of urban imagery.

For $a\in\mathcal{A}$ with satellite imagery $I^{\text{SI}}_{a}$, the visual
representation can be calculated as
$\bm{I}^{\text{SI}}_{a}=\text{ResNet}(I^{\text{SI}}_{a})$. As for street view
imagery
$I^{\text{SV}}_{a}=\\{I^{\text{SV}}_{a,1},\\!\cdots\\!,I^{\text{SV}}_{a,n}\\}$,
the visual representation is calculated by average pooling on street view
images therein:

(2)
$\displaystyle\bm{I}^{\text{SV}}_{a}=\frac{1}{n}\sum_{i=1}^{n}\text{ResNet}(I^{\text{SV}}_{a,i}).$

Thus, let $f^{\text{Image}}(\cdot)$ denote the visual encoder designed above,
and the urban imagery representation can be obtained by
$\bm{I}_{a}=f^{\text{Image}}(I_{a})$ with
$I_{a}=I^{\text{SI}}_{a}/I^{\text{SV}}_{a}$ and
$\bm{I}_{a}=\bm{I}^{\text{SI}}_{a}/\bm{I}^{\text{SV}}_{a}$.

###  4.4. Contrastive Loss Design & Optimization

Motivated by cross-modality based contrastive learning between image and text
modalities (Zhang et al., 2020; Radford et al., 2021), we design a novel
image-KG contrastive loss for knowledge infusion. The core insight here is
that both semantic representation (KG embedding) and visual representation
(urban imagery representation) of a region should be close to each other.

First, for better representation quality, we introduce two independent
projection heads $g^{\text{KG}}(\cdot)$ and $g^{\text{Image}}(\cdot)$ after
semantic encoder and visual encoder, respectively, as validated in empirical
studies (Chen et al., 2020; Bjorck et al., 2021). The corresponding outputs of
$\tilde{\bm{e}}_{a}$ and $\tilde{\bm{I}}_{a}$ for $a\in\mathcal{A}$ can be
expressed as:

(3)
$\displaystyle\tilde{\bm{e}}_{a}=g^{\text{KG}}(\bm{e}_{a})=\bm{W}^{\text{KG}}_{2}\text{ReLU}(\bm{W}^{\text{KG}}_{1}\bm{e}_{a})$
(4)
$\displaystyle\tilde{\bm{I}}_{a}=g^{\text{Image}}(\bm{I}_{a})=\bm{W}^{\text{Image}}_{2}\text{ReLU}(\bm{W}^{\text{Image}}_{1}\bm{I}_{a}),$

where four projection matrices are used to project representations for both
modalities from their encoder space to the same space for contrastive
learning.

Moreover, following the core insight above, we extend the traditional InfoNCE
loss (Oord et al., 2018) to image-KG contrastive loss, and the loss function
for $a\in\mathcal{A}$ is expressed as:

$\displaystyle\mathcal{L}_{a}$
$\displaystyle=\mathcal{L}_{a}^{\text{Image}\rightarrow\text{KG}}+\mathcal{L}_{a}^{\text{KG}\rightarrow\text{Image}}$
(5)
$\displaystyle=-\log\\!\frac{\exp(\text{sim}(\tilde{\bm{I}}_{a},\tilde{\bm{e}}_{a}))}{\sum_{i=1}^{m}\exp(\text{sim}(\tilde{\bm{I}}_{a},\tilde{\bm{e}}_{i}))}\\!-\\!\log\\!\frac{\exp(\text{sim}(\tilde{\bm{e}}_{a},\tilde{\bm{I}}_{a}))}{\sum_{i=1}^{m}\exp(\text{sim}(\tilde{\bm{e}}_{a},\tilde{\bm{I}}_{i}))},$

where the loss is computed in a minibatch of $m$ samples, and
$\text{sim}(\cdot)$ represents the inner product.
$\mathcal{L}_{a}^{\text{Image}\rightarrow\text{KG}}$ and
$\mathcal{L}_{a}^{\text{KG}\rightarrow\text{Image}}$ are image-to-KG and KG-
to-image contrastive losses, respectively, which maximally preserve the mutual
information between image-KG pairs. Unlike existing urban imagery-based
socioeconomic prediction studies using image view-based contrastive loss in
the same modality (Xi et al., 2022; Li et al., 2022), our proposed image-KG
contrastive loss is based on cross modalities of inputs, which successfully
infuses the comprehensive knowledge captured in region embeddings into urban
imagery representations.

By optimizing the image-KG contrastive loss on the whole data, we obtain
knowledge-infused urban imagery representations $\bm{I}_{a\in\mathcal{A}}$
from the visual encoder, which are further fed into the regression module of
multi-layer perceptron (MLP) for socioeconomic indicator training and
prediction.

##  5\. Experiments and Results

###  5.1. Experimental Setup

####  5.1.1. Datasets

We collect three datasets with urban imagery and socioeconomic indicator data
for evaluation: Beijing (BJ), Shanghai (SH) and New York (NY). Regions in
Beijing and Shanghai are partitioned by road network, while regions in New
York are Census Block Groups (CBGs) used by US Census Bureau. The
256$\times$256-pixel satellite images with about 4.7 m-resolution are obtained
from ArcGIS, which are further merged along irregular region boundaries for
input satellite images. The 1024$\times$512-pixel street view images in
Beijing and Shanghai as well as the 512$\times$512-pixel street view images in
New York are obtained from Baidu Map API and Google Street API, respectively.

As for socioeconomic indicator data, Beijing dataset includes (1) Pop.:
population data from WorldPop, (2) Econ.: economic activity data from (Dong et
al., 2021), (3) Rest.: restaurant business (takeaway order data) and (4)
Consp.: consumption data from a life service platform, while Shanghai dataset
includes (1) Pop.: population and (2) Econ.: economic activity data from same
sources. New York dataset includes (1) Pop.: population and (2) Edu.:
education data from SafeGraph, and (3) Crime: crime data from NYC Open Data.
The UrbanKG data for three datasets are from (Liu et al., 2021a; Wang et al.,
2020b), and business knowledge related relations are omitted in New York
dataset due to a lack of source data. All socioeconomic indicators are
converted into logarithmic scale, i.e. $y=\ln(y_{\text{raw}}+1)$. Besides,
regions in datasets with over 40 street view images are selected and randomly
split into train/valid/test sets by a proportion of 6:2:2 in the socioeconomic
prediction step. Table 2 summarizes dataset statistics and details are
provided in Section A.

Table 2. Dataset Statistics. Dataset | #Region | #SV | #SI | $|\mathcal{E}|$ | $|\mathcal{R}|$ | $|\mathcal{F}|$
---|---|---|---|---|---|---
Beijing | 789 | 31,560 | 18,289 | 36,752 | 10 | 188,985
Shanghai | 1,553 | 62,120 | 5,904 | 58,145 | 10 | 363,159
New York | 1,142 | 45,680 | 1,560 | 87,020 | 6 | 357,464
Table 3. Satellite imagery-based socioeconomic prediction results on three datasets. Best results are in bold and the best results are underlined. The last row shows relative improvement in percentage. Dataset | Beijing | Shanghai | New York
---|---|---|---
Model | Pop. |  | Econ.
---
| Rest.
---
Consp. | Pop. |  | Econ.
---
Pop. | Edu. | Crime
$R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE
ResNet-18 | 0.277 | 0.887 | 0.168 | 1.465 | 0.146 | 2.569 | 0.125 | 3.435 | 0.007 | 1.027 | 0.082 | 1.674 | -0.404 | 0.788 | 0.518 | 0.085 | 0.324 | 0.751
Tile2Vec | 0.274 | 0.888 | 0.092 | 1.531 | 0.108 | 2.626 | 0.074 | 3.534 | 0.125 | 1.041 | 0.065 | 1.689 | 0.143 | 0.615 | 0.533 | 0.084 | 0.382 | 0.718
READ | 0.300 | 0.872 | 0.173 | 1.461 | 0.222 | 2.451 | 0.213 | 3.258 | 0.154 | 0.949 | 0.097 | 1.660 | -0.034 | 0.676 | 0.534 | 0.084 | 0.413 | 0.700
PG-SimCLR | 0.356 | 0.837 | 0.361 | 1.285 | 0.275 | 2.368 | 0.269 | 3.140 | 0.307 | 0.858 | 0.166 | 1.596 | -0.223 | 0.735 | 0.622 | 0.075 | 0.434 | 0.687
KnowCL | 0.479 | 0.752 | 0.532 | 1.100 | 0.493 | 1.979 | 0.443 | 2.741 | 0.424 | 0.783 | 0.325 | 1.436 | 0.153 | 0.612 | 0.658 | 0.042 | 0.536 | 0.622
Improv. | 34.5% | 10.2% | 47.3% | 14.4% | 79.3% | 16.4% | 64.7% | 12.7% | 38.1% | 8.7% | 95.8% | 10.0% | 7.0% | 0.5% | 5.8% | 44.0% | 23.5% | 9.5%
Table 4. Street view imagery-based socioeconomic prediction results on three datasets. Best results are in bold and the best results are underlined. The last row shows relative improvement in percentage. Dataset | Beijing | Shanghai | New York
---|---|---|---
Model | Pop. |  | Econ.
---
| Rest.
---
Consp. | Pop. |  | Econ.
---
Pop. | Edu. | Crime
$R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE
ResNet-18 | 0.085 | 0.997 | 0.215 | 1.423 | 0.262 | 2.388 | 0.257 | 3.166 | 0.046 | 1.007 | 0.033 | 1.718 | 0.151 | 0.612 | 0.402 | 0.095 | 0.340 | 0.742
Urban2Vec | 0.026 | 1.029 | 0.059 | 1.559 | 0.094 | 2.646 | 0.103 | 3.478 | 0.012 | 1.025 | 0.007 | 1.741 | 0.046 | 0.649 | 0.232 | 0.107 | 0.020 | 0.904
SceneParse | 0.073 | 1.004 | 0.157 | 1.476 | 0.183 | 2.512 | 0.193 | 3.299 | 0.058 | 1.001 | 0.049 | 1.704 | 0.154 | 0.611 | 0.426 | 0.093 | 0.222 | 0.806
PG-SimCLR | 0.237 | 0.911 | 0.288 | 1.356 | 0.409 | 2.136 | 0.439 | 2.750 | 0.015 | 1.023 | 0.112 | 1.646 | 0.283 | 0.563 | 0.569 | 0.080 | 0.482 | 0.657
KnowCL | 0.416 | 0.796 | 0.557 | 1.069 | 0.470 | 2.024 | 0.449 | 2.725 | 0.359 | 0.826 | 0.281 | 1.482 | 0.377 | 0.524 | 0.586 | 0.079 | 0.552 | 0.612
Improv. | 75.6% | 12.6% | 44.3% | 21.2% | 14.9% | 5.2% | 2.3% | 1.0% | 519.0% | 17.5% | 150.9% | 10.0% | 33.2% | 6.9% | 3.0% | 1.3% | 14.5% | 6.8%

####  5.1.2. Baselines

We compare our model with several baselines in urban imagery-based
socioeconomic prediction studies. The satellite imagery-based baselines
include:

  * •

Tile2Vec (Jean et al., 2019). Tile2Vec uses the triplet loss to minimize
visual representations of spatially near satellite images and maximize those
of distant pairs.

  * •

READ (Han et al., 2020a). READ uses limited label data to train a teacher-
student network with satellite imagery. The pre-trained model in original
paper is used for comparison.

The street view imagery-based baselines includes:

  * •

Urban2Vec (Wang et al., 2020a). Urban2Vec follows the similar design with
Tile2Vec but focuses on street view imagery.

  * •

SceneParse (Zhou et al., 2017). We use this scene parsing model to extract
street view imagery representations following (Li et al., 2022; Lee et al.,
2021).

We also apply two baselines for both satellite and street view imagery-based
socioeconomic prediction:

  * •

ResNet-18 (He et al., 2016). ResNet-18 is pre-trained on ImageNet (Deng et
al., 2009), which is a backbone adopted in several related studies, and thus
selected for comparison in both satellite and street view imagery.

  * •

PG-SimCLR (Xi et al., 2022). PG-SimCLR originally employs SimCLR (Chen et al.,
2020) for satellite imagery-based socioeconomic prediction, with spatiality
and POI category distribution considered in similarity metric design. We
select it for both satellite and street view imagery cases considering its
competitive performance.

We implement the baselines following reported settings or using pre-trained
models in their original papers, and the obtained urban imagery
representations are fed into the MLP-based socioeconomic indicator regression
module for training and prediction.

####  5.1.3. Metrics & Implementation

We adopt the widely used rooted mean squared error (RMSE) and coefficient of
determination ($R^{2}$) (Xi et al., 2022; Jean et al., 2016; Han et al.,
2020a) for evaluation metrics. For the implementation, ResNet-18 (He et al.,
2016) and CompGCN (Vashishth et al., 2019) are adopted for visual and semantic
encoders, respectively. We select Adam optimizer for parameter learning. In
the contrastive learning step, we set the KG embedding dimension as 64 while
the number of GCN layers is selected from $\\{1,2,3,4\\}$. The learning rate
is set as 0.0003. In the socioeconomic prediction step, for each region, the
learnt single urban imagery representation vector is used to predict various
socioeconomic indicators with the learning rate and dropout searched from
$\\{0.0005,0.001,0.005\\}$ and $\\{0.1,0.3,0.5\\}$. Besides, we randomly
select 10 street view images for each region in the main experiment. The
implementation codes are available at the link444https://github.com/tsinghua-
fib-lab/UrbanKG-KnowCL.

###  5.2. Performance Comparison

We evaluate the satellite imagery-based socioeconomic prediction performance
in Table 3. Results on three datasets across six types of socioeconomic
indicators demonstrate the superiority of our proposed KnowCL model, which
improves the best baseline (PG-SimCLR) by 7%-79% on $R^{2}$ in all cases.
Especially, KnowCL achieves the significant performance improvements owing to
the comprehensive knowledge infused by the cross-modality based contrastive
learning. As for the performance comparison with contrastive learning based
models like Tile2Vec and PG-SimCLR, the results show that introducing more
knowledge can bring better performance. For example, Tile2Vec only considers
spatiality knowledge in similarity metric design while PG-SimCLR further
considers function knowledge, leading to over 15% improvements on $R^{2}$ on
average. Besides, traditional models focus on satellite images in grid shape,
failing to extract high-quality visual representations for the more practical
case of irregular shape partitioned by road network.

As for the street view imagery-based socioeconomic prediction performance in
Table 4, KnowCL also achieves state-of-the-art results, which further validate
the effectiveness and robustness. Limited by the repetitive street view images
collected in Shanghai dataset, all baselines perform poorly across population
and economic activeness prediction tasks therein, while KnowCL leverages
UrbanKGs for informative representation learning from urban imagery with
competitive performance achieved.

According to the absolute performance in Table 3 and Table 4, the
socioeconomic indicators of different cities show diverse preference to urban
imagery, e.g., KnowCL with satellite imagery obtains the best absolute
performance for population prediction in Beijing and Shanghai, while the
street view imagery becomes the better choice for population and crime
prediction in New York. This phenomenon is mainly determined by city
structures and socioeconomic indicator characteristics. Different from complex
city structures in Beijing and Shanghai, New York follows the grid layout with
block regions in similar shapes, which provides limited information for
population estimation. Besides, the street view imagery can provide an
internal view for urban environment safety perception, as validated in (Naik
et al., 2014). Such results further indicate that both satellite and street
view imagery can provide complementary information to each other, and our
proposed KnowCL model can fully exploit the value of urban imagery, which is
quite essential for urban environment perception and SDG monitoring.

(a) Satellite Imagery (b) Street View Imagery

Figure 3. Predicted population versus true population across all regions on
Beijing dataset. Blue line is at 45°. $R^{2}$ and $R^{2}_{a}$ correspond to
the results of testing regions (red dots) and all regions (red and blue dots),
respectively

To further analyze the predictive power of our proposed KnowCL model, in
Figure 3, we compare the predicted and true population for all regions in
Beijing dataset, and results for other datasets are provided in Section B. The
results show that KnowCL can well replicate the population of most regions
(see the dots along 45° line) via urban imagery, explaining 52%-63% of the
variation in population on two datasets.

###  5.3. Ablation Study

####  5.3.1. Effectiveness of Knowledge Identification

To validate the effectiveness of identified knowledge in UrbanKG, Figure 4
presents the performance comparison of UrbanKG without certain type of
knowledge. We select Beijing and New York datasets for evaluation, on which
satellite and street view imagery inputs achieve the best absolute
performance, respectively. The business knowledge on New York dataset is not
provided in UrbanKG and thus not reported.

(a) Satellite Imagery (b) Street View Imagery

Figure 4. Performance comparison of different identified knowledge on Beijing
and New York datasets with satellite and street view imagery, respectively.

All four types of semantic knowledge identified by UrbanKG are essential for
socioeconomic prediction, according to the findings in Figure 4. Particularly,
the knowledge that is hardly captured by urban imagery is more important,
e.g., the mobility knowledge of crowd flow transitions between regions brings
5%-30% gains for predicting all socioeconomic indicators, because both
satellite and street view imagery cannot capture such dynamic information
without additional knowledge infused. Additionally, the impacts of various
types of semantic knowledge vary to socioeconomic indicators. For example, in
New York dataset, the education indicator is highly correlated with function
knowledge while the population indicator prefers to spatiality knowledge,
which enlightens us to identify comprehensive knowledge for a broader urban
imagery-based socioeconomic prediction with more indicators considered.

####  5.3.2. Effectiveness of Knowledge Infusion

A major novelty of this paper is introducing the cross-modality based
contrastive learning with the image-KG contrastive loss for knowledge
infusion, which is different from the single-modality based ones in existing
studies (Jean et al., 2019; Xi et al., 2022). To validate the effectiveness,
we develop a direct image-image contrastive loss for knowledge infusion, which
is similar to PG-SimCLR (Xi et al., 2022), termed as KG-SimCLR. Specifically,
KG-SimCLR calculates KG embedding similarity for positive region pairs, and
requires their associated urban images to be closer in visual representation
space.

Table 5. Performance comparison $R^{2}$ of different knowledge infusion ways on Beijing and New York datasets. |  | Beijing | New York
---|---|---|---
Model | Pop. | Econ. | Rest. | Consp. | Pop. | Edu. | Crime
SI | KG-SimCLR | 0.272 | 0.197 | 0.192 | 0.175 | -0.302 | 0.555 | 0.341
KnowCL | 0.479 | 0.532 | 0.493 | 0.443 | 0.153 | 0.658 | 0.536
SV | KG-SimCLR | 0.209 | 0.229 | 0.299 | 0.329 | 0.053 | 0.382 | 0.294
KnowCL | 0.416 | 0.557 | 0.470 | 0.449 | 0.377 | 0.586 | 0.552
Figure 5. Most similar urban imagery matching between Beijing and Shanghai
datasets via learnt urban imagery representations by KnowCL. The population
indicator and cosine similarity are presented below the images. Satellite
images might be in irregular shape due to the shape of associated regions.

(a) Satellite Imagery (b) Street View Imagery

Figure 6. The $R^{2}$ for the transferability test on satellite and street
view imagery-based population prediction.

Table 5 presents the performance comparison between KG-SimCLR and KnowCL with
different urban imagery inputs. The significant performance gaps between two
models on both satellite and street view imagery-based socioeconomic
prediction indicate that simply using existing image view-based contrastive
loss cannot achieve effective knowledge infusion. Especially, directly
modeling the comprehensive knowledge in KG as a similarity metric provides a
quite weak self-supervision signal for visual representation learning, while
our proposed cross-modality based contrastive learning framework infuses such
knowledge via similarity matching in representation space. Overall, the
ablation studies demonstrate the effectiveness of our proposed knowledge
infusion design and can potentially apply in various urban imagery-based
research.

###  5.4. Transferability Study

####  5.4.1. Prediction Performance Across Cities/Countries

The experiment results above validate the effectiveness of UrbanKG, which
however might be not available in underdeveloped and developing
cities/countries due to data deficiency. Thus, here we investigate the
practical case of socioeconomic prediction in transfer setting (Park et al.,
2022): Given the visual encoder of a KnowCL model trained on a source city
with urban imagery and UrbanKG data, we apply it for socioeconomic prediction
in target cities where only urban imagery data are available. The
transferability task checks whether KnowCL infuses shared knowledge across
cities into the visual encoder for urban imagery-based socioeconomic
prediction.

We vary the source-target city pairs and report the population prediction
performance in Figure 6, where both satellite imagery and street view imagery
are evaluated. Here 40 street view images for each region in off-diagonal
transfer experiments are used for robust performance. According to the
results, the diagonal line shows the highest correlation due to the same city
transferred from the source to the target. Moreover, compared with baselines
trained and evaluated on the same dataset in Table 3 and Table 4, KnowCL
achieves competitive transfer performance for both satellite and street view
imagery-based population prediction in Bejing and New York datasets, as
validated by similar scatter sizes in each row. For example, SH$\rightarrow$BJ
transfer experiment achieves a $R^{2}$ of 0.373 compared with 0.356 of the
best baseline (PG-SimCLR) achieved in non-transfer setting. Such results
validate the transferability of our proposed KnowCL model for socioeconomic
prediction across cities/countries, which mainly owes to the shared knowledge
identified in UrbanKG and infused in visual encoder by cross-modality based
contrastive learning. Hence, pre-trained KnowCL model can be leveraged for
socioeconomic prediction in cities without UrbanKG. Besides, we also
investigate the transferability of the best baseline PG-SimCLR in Section B,
which is less competitive due to limited knowledge considered.

####  5.4.2. Visual Analogies Across Cities

We also investigate the visual similarity between urban imagery across cities.
Specifically, given an urban image in source city, we compute the cosine
similarity between its visual representation and all visual representations in
another city, and select the most similar ones for comparison (Wang et al.,
2020a), as shown in Figure 5. As for the satellite imagery matching in Figure
5(a), similar regions share the similar distribution of buildings as well as
populations. On the other hand, the street view imagery matching in Figure
5(b) successfully identifies regions with similar physical appearance and
populations. Thus, the knowledge-infused urban imagery representations capture
not only visual features but also socioeconomic information associated with
regions.

##  6\. Conclusion

In this paper, we present a novel approach to make predictions on population,
economic activity, consumption, education, and public safety indicators from
web-collected urban imagery covering both satellite and street view imagery.
Our proposed knowledge-infused contrastive learning model KnowCL is built upon
the comprehensive knowledge identified by urban knowledge graph, and further
designs an image-KG contrastive loss for effective knowledge infusion into
urban imagery representations. KnowCL is the first solution that introduces
the knowledge graph and the cross-modality based contrastive learning
framework for urban imagery-based socioeconomic prediction. Extensive
experiments validate the model effectiveness and transferability in different
cities across several socioeconomic indicators.

We demonstrated model’s potential in socioeconomic prediction with ubiquitous
urban imagery, which is of great importance to sustainable development in
data-poor regions and countries. Although our model outperforms baselines, the
results may be less interpretable, so in future work we will consider
exploring in-depth the semantics of UrbanKG for interpretability.

###### Acknowledgements.

This work was supported in part by the National Key Research and Development
Program of China under 2020AAA0106000, the National Nature Science Foundation
of China under 61972223, 61971267, U1936217.

## References

  * (1)
  * Abitbol and Karsai (2020) Jacob Levy Abitbol and Marton Karsai. 2020.  Interpretable Socioeconomic Status Inference from Aerial Imagery through Urban Patterns.  _Nature Machine Intelligence_ 2, 11 (2020), 684–692.
  * Ahn et al. (2020) Donghyun Ahn, Meeyoung Cha, Sungwon Han, Jihee Kim, Susang Lee, Sangyoon Park, Sungwon Park, Hyunjoo Yang, and Jeasurk Yang. 2020.  Teaching Machines to Measure Economic Activities from Satellite Images: Challenges and Solutions.  (2020).
  * Balažević et al. (2019) Ivana Balažević, Carl Allen, and Timothy M Hospedales. 2019\.  TuckER: Tensor Factorization for Knowledge Graph Completion. In _EMNLP_. 5185–5194.
  * Bank (2022) The World Bank. 2022\.  Urban Development.  https://www.worldbank.org/en/topic/urbandevelopment/overview.  Accessed: 2022-11-01.
  * Bjorck et al. (2021) Johan Bjorck, Brendan H Rappazzo, Qinru Shi, Carrie Brown-Lima, Jennifer Dean, Angela Fuller, and Carla Gomes. 2021.  Accelerating Ecological Sciences from Above: Spatial Contrastive Learning for Remote Sensing. In _AAAI_ , Vol. 35. 14711–14720.
  * Burke et al. (2021) Marshall Burke, Anne Driscoll, David B Lobell, and Stefano Ermon. 2021.  Using Satellite Imagery to Understand and Promote Sustainable Development.  _Science_ 371, 6535 (2021), eabe8628.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A Simple Framework for Contrastive Learning of Visual Representations. In _ICML_. 1597–1607.
  * Corsi et al. (2012) Daniel J Corsi, Melissa Neuman, Jocelyn E Finlay, and SV Subramanian. 2012.  Demographic and Health Surveys: A Profile.  _International Journal of Epidemiology_ 41, 6 (2012), 1602–1613.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A Large-scale Hierarchical Image Database. In _CVPR_. 248–255.
  * Dong et al. (2021) Lei Dong, Xiaohui Yuan, Meng Li, Carlo Ratti, and Yu Liu. 2021.  A Gridded Establishment Dataset as A Proxy for Economic Activity in China.  _Scientific Data_ 8, 1 (2021), 1–9.
  * Gebru et al. (2017) Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, and Li Fei-Fei. 2017.  Using Deep Learning and Google Street View to Estimate the Demographic Makeup of Neighborhoods across the United States.  _PNAS_ 114, 50 (2017), 13108–13113.
  * Han et al. (2020a) Sungwon Han, Donghyun Ahn, Hyunji Cha, Jeasurk Yang, Sungwon Park, and Meeyoung Cha. 2020a.  Lightweight and Robust Representation of Economic Scales from Satellite Imagery. In _AAAI_ , Vol. 34. 428–436.
  * Han et al. (2020b) Sungwon Han, Donghyun Ahn, Sungwon Park, Jeasurk Yang, Susang Lee, Jihee Kim, Hyunjoo Yang, Sangyoon Park, and Meeyoung Cha. 2020b.  Learning to Score Economic Development from Satellite Imagery. In _KDD_. 2970–2979.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _CVPR_. 770–778.
  * He et al. (2018) Zhiyuan He, Su Yang, Weishan Zhang, and Jiulong Zhang. 2018\.  Perceiving Commerial Activeness Over Satellite Images. In _WWW_. 387–394.
  * Hogan et al. (2021) Aidan Hogan, Eva Blomqvist, Michael Cochez, Claudia d’Amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sebastian Neumaier, et al. 2021\.  Knowledge Graphs.  _ACM Comput. Surv._ 54, 4 (2021), 1–37.
  * Jean et al. (2016) Neal Jean, Marshall Burke, Michael Xie, W Matthew Davis, David B Lobell, and Stefano Ermon. 2016\.  Combining Satellite Imagery and Machine Learning to Predict Poverty.  _Science_ 353, 6301 (2016), 790–794.
  * Jean et al. (2019) Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, and Stefano Ermon. 2019\.  Tile2vec: Unsupervised Representation Learning for Spatially Distributed Data. In _AAAI_ , Vol. 33. 3967–3974.
  * Jing and Tian (2020) Longlong Jing and Yingli Tian. 2020.  Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey.  _IEEE TPAMI_ 43, 11 (2020), 4037–4058.
  * Kang et al. (2020) Jian Kang, Ruben Fernandez-Beltran, Puhong Duan, Sicong Liu, and Antonio J Plaza. 2020\.  Deep Unsupervised Embedding for Remotely Sensed Images based on Spatially Augmented Momentum Contrast.  _IEEE Transactions on Geoscience and Remote Sensing_ 59, 3 (2020), 2598–2610.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017\.  Imagenet Classification with Deep Convolutional Neural Networks.  _Commun. ACM_ 60, 6 (2017), 84–90.
  * Law et al. (2019) Stephen Law, Brooks Paige, and Chris Russell. 2019\.  Take a Look Around: Using Street View and Satellite Images to Estimate House Prices.  _ACM TIST_ 10, 5 (2019), 1–19.
  * Lee et al. (2021) Jihyeon Lee, Dylan Grosz, Burak Uzkent, Sicheng Zeng, Marshall Burke, David Lobell, and Stefano Ermon. 2021.  Predicting Livelihood Indicators from Community-Generated Street-Level Imagery. In _AAAI_ , Vol. 35. 268–276.
  * Li et al. (2020) Shuangli Li, Jingbo Zhou, Tong Xu, Hao Liu, Xinjiang Lu, and Hui Xiong. 2020\.  Competitive Analysis for Points of Interest. In _KDD_. 1265–1274.
  * Li et al. (2022) Tong Li, Shiduo Xin, Yanxin Xi, Sasu Tarkoma, Pan Hui, and Yong Li. 2022\.  Predicting Multi-level Socioeconomic Indicators from Structural Urban Imagery. In _CIKM_. 3282–3291.
  * Liu et al. (2021b) Jia Liu, Tianrui Li, Shenggong Ji, Peng Xie, Shengdong Du, Fei Teng, and Junbo Zhang. 2021b.  Urban Flow Pattern Mining based on Multi-source Heterogeneous Data Fusion and Knowledge Graph Embedding.  _IEEE TKDE_ (2021).
  * Liu et al. (2021c) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021c.  Self-supervised Learning: Generative or Contrastive.  _IEEE TKDE_ (2021).
  * Liu et al. (2021a) Yu Liu, Jingtao Ding, and Yong Li. 2021a.  Knowledge-driven Site Selection via Urban Knowledge Graph.  _arXiv preprint arXiv:2111.00787_ (2021).
  * Liu et al. (2022) Yu Liu, Jingtao Ding, and Yong Li. 2022.  Developing Knowledge Graph based System for Urban Computing. In _ACM SIGSPATIAL Geospatial Knowledge Graphs Workshop_. 3–7.
  * Miller (2004) Harvey J Miller. 2004\.  Tobler’s First Law and Spatial Analysis.  _Annals of the Association of American Geographers_ 94, 2 (2004), 284–289.
  * Naik et al. (2014) Nikhil Naik, Jade Philipoom, Ramesh Raskar, and César Hidalgo. 2014.  Streetscore-predicting the Perceived Safety of One Million Streetscapes. In _CVPR_. 779–785.
  * of Economic and Affairs (2018) UN Department of Economic and Social Affairs. 2018.  World Urbanization Prospects: The 2018 Revision.
  * of Economic and Affairs (2022) UN Department of Economic and Social Affairs. 2022.  The Sustainable Development Goals Report 2022.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation Learning with Contrastive Predictive Coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Park et al. (2022) Sungwon Park, Sungwon Han, Donghyun Ahn, Jaeyeon Kim, Jeasurk Yang, Susang Lee, Seunghoon Hong, Jihee Kim, Sangyoon Park, Hyunjoo Yang, et al. 2022\.  Learning Economic Indicators by Aggregating Multi-level Geospatial Information. In _AAAI_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning Transferable Visual Models from Natural Language Supervision. In _ICML_. 8748–8763.
  * Rolf et al. (2021) Esther Rolf, Jonathan Proctor, Tamma Carleton, Ian Bolliger, Vaishaal Shankar, Miyabi Ishihara, Benjamin Recht, and Solomon Hsiang. 2021\.  A Generalizable and Accessible Approach to Machine Learning with Global Satellite Imagery.  _Nature Communications_ 12, 1 (2021), 1–11.
  * Suel et al. (2019) Esra Suel, John W Polak, James E Bennett, and Majid Ezzati. 2019\.  Measuring Social, Environmental and Health Inequalities Using Deep Learning and Street Imagery.  _Scientific Reports_ 9, 1 (2019), 1–10.
  * Vashishth et al. (2019) Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019.  Composition-based Multi-relational Graph Convolutional Networks. In _ICLR_.
  * Wang et al. (2021) Huandong Wang, Qiaohong Yu, Yu Liu, Depeng Jin, and Yong Li. 2021.  Spatio-Temporal Urban Knowledge Graph Enabled Mobility Prediction.  _ACM UbiComp_ 5, 4 (2021), 1–24.
  * Wang et al. (2020b) Pengyang Wang, Kunpeng Liu, Lu Jiang, Xiaolin Li, and Yanjie Fu. 2020b.  Incremental Mobile User Profiling: Reinforcement Learning with Spatial Knowledge Graph for Modeling Event Streams. In _KDD_. 853–861.
  * Wang et al. (2017) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017\.  Knowledge Graph Embedding: A Survey of Approaches and Applications.  _IEEE TKDE_ 29, 12 (2017), 2724–2743.
  * Wang et al. (2018) Wenshan Wang, Su Yang, Zhiyuan He, Minjie Wang, Jiulong Zhang, and Weishan Zhang. 2018\.  Urban Perception of Commercial Activeness from Satellite Images and Streetscapes. In _WWW_. 647–654.
  * Wang et al. (2022) Yi Wang, Conrad Albrecht, Nassim Ait Ali Braham, Lichao Mou, and Xiaoxiang Zhu. 2022.  Self-Supervised Learning in Remote Sensing: A Review.  _IEEE Geoscience and Remote Sensing Magazine_ (2022).
  * Wang et al. (2020a) Zhecheng Wang, Haoyuan Li, and Ram Rajagopal. 2020a.  Urban2vec: Incorporating Street View Imagery and POIs for Multi-modal Urban Neighborhood Embedding. In _AAAI_ , Vol. 34. 1013–1020.
  * Wu et al. (2022) Shangbin Wu, Xu Yan, Xiaoliang Fan, Shirui Pan, Shichao Zhu, Chuanpan Zheng, Ming Cheng, and Cheng Wang. 2022\.  Multi-Graph Fusion Networks for Urban Region Embedding.  _arXiv preprint arXiv:2201.09760_ (2022).
  * Xi et al. (2022) Yanxin Xi, Tong Li, Huandong Wang, Yong Li, Sasu Tarkoma, and Pan Hui. 2022\.  Beyond the First Law of Geography: Learning Representations of Satellite Imagery by Leveraging Point-of-Interests. In _WWW_. 3308–3316.
  * Yeh et al. (2020) Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermon, and Marshall Burke. 2020\.  Using Publicly Available Satellite Imagery and Deep Learning to Understand Economic Well-being in Africa.  _Nature Communications_ 11, 1 (2020), 1–11.
  * Zhang et al. (2021) Mingyang Zhang, Tong Li, Yong Li, and Pan Hui. 2021\.  Multi-view Joint Graph Representation Learning for Urban Region Embedding. In _IJCAI_. 4431–4437.
  * Zhang et al. (2020) Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. 2020.  Contrastive Learning of Medical Visual Representations from Paired Images and Text.  _arXiv preprint arXiv:2010.00747_ (2020).
  * Zheng et al. (2014) Yu Zheng, Licia Capra, Ouri Wolfson, and Hai Yang. 2014\.  Urban Computing: Concepts, Methodologies, and Applications.  _ACM TIST_ 5, 3 (2014), 1–55.
  * Zhou et al. (2017) Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017\.  Scene Parsing through ADE20K Dataset. In _CVPR_. 633–641.
  * Zhuang et al. (2017) Chenyi Zhuang, Nicholas Jing Yuan, Ruihua Song, Xing Xie, and Qiang Ma. 2017\.  Understanding People Lifestyles: Construction of Urban Movement Knowledge Graph from GPS Trajectory. In _IJCAI_. 3616–3623.

##  Appendix A Details of Dataset

###  A.1. UrbanKG Construction

Here we introduce the details of urbanKG construction. For region entities in
UrbanKG for Beijing and Shanghai datasets, we partition the city into multiple
regions by the road network, which are shown in Figure 7. The region entities
in New York dataset follows the CBG division by US Census Bureau, whose
visualization can be referred to the official
link555https://data.cityofnewyork.us/City-Government/2010-Census-
Blocks/v2h8-6mxf. Each region entity is provided with a sequence of longitude-
latitude pairs
$L_{a}=\\{(lng_{a}^{1},lat_{a}^{1}),\cdots,(lng_{a}^{k},lat_{a}^{k})\\}$ as
region boundary. POI entities and business center entities are provided with
location information of longitude-latitude pairs like
$l_{i}=(lng_{i},lat_{i})$. The category entities are POI properties identified
by experts, e.g., food, shopping, accommodation, business, residence,
education, etc.

(a) Beijing

(b) Shanghai

Figure 7. Visualization of region entities in UrbanKG for Beijing and Shanghai
datasets.

Based on the entities above, the relational links defined in Table 1 can be
extracted as follows.

  * •

_borderBy._ Given two regions $a,b$, they are connected by _borderBy_ if
$|L_{a}\cap L_{b}|>0$, i.e., sharing the same boundary points.

  * •

_nearBy._ Given two regions $a,b$, they are connected by _nearBy_ if
$\|\bar{L}_{a}-\bar{L}_{b}\|\leq 1km$, where $\bar{L}_{a},\bar{L}_{b}$ are
center location of regions.

  * •

_locateAt._ Given a POI $p$ and a region $a$, they are connected by _locateAt_
if $l_{p}$ is in the closure by region boundary $L_{a}$.

  * •

_flowTransition._ Given two regions $a,b$, they are connected by
_flowTransition_ if the aggregated flow transition between two regions exceeds
the threshold.

  * •

_similarFunction._ Given two regions $a,b$ and the category distribution
vectors of POIs therein $\bm{z}_{a},\bm{z}_{b}$, they are connected by
_similarrFunction_ if $cos(\bm{z}_{a},\bm{z}_{b})\geq 0.95$ with cosine
similarity.

  * •

_coCheckin._ Given two POIs $p_{1},p_{2}$, they are connected by _coCheckin_
if the number of records that consecutively visit $p_{1}$ and $p_{2}$ exceeds
the threshold.

  * •

_cateOf._ A POI is connected to its associated category by _cateOf_.

  * •

_provideService._ Given a region $a$ and a business center $bc$, they are
connected by _provideService_ if $\|\bar{L}_{a}-l_{bc}\|\leq 3km$.

  * •

_belongTo._ Given a POI $p$ and a business center $bc$, they are connected by
_belongTo_ if $\|l_{p}-l_{bc}\|\leq 3km$.

  * •

_competitive._ Given two POIs $p_{1},p_{2}$, they are connected by
_competitive_ if $\|l_{p_{1}}-l_{p_{2}}\|\leq 500m$ and they are in the same
category.

(a) Satellite Imagery (SH) (b) Street View Imagery (SH) (c) Satellite Imagery
(NY) (d) Street View Imagery (NY)

Figure 8. Predicted population versus true population across all regions on
Shanghai and New York datasets. Blue line is at 45°. $R^{2}$ and $R^{2}_{a}$
correspond the results of testing regions (red dots) and all regions (red and
blue dots), respectively

(a) Satellite (BJ) (b) Street View (BJ) (c) Satellite (SH) (d) Street View
(SH)

(e) Satellite (NY) (f) Street View (NY)

Figure 9. Visualization of urban imagery based on PCA algorithm, where dot
color represents the value of corresponding population to the urban imagery.

(a) Satellite Imagery (b) Street View Imagery

Figure 10. The $R^{2}$ for the transferability test of PG-SimCLR on satellite
and street view imagery-based population prediction. The visual encoder is
trained on one source city and then evaluated on other target cities.

###  A.2. Data Sources & Preprocessing

The data sources in our experiments and their links are provided as follows.

  * •

Satellite imagery data. ArGIS, https://geoenrich.arcgis.com/.

  * •

Street view imagery data in Beijing & Shanghai. Baidu Map API,
http://api.map.baidu.com.

  * •

Street view imagery data in New York. Google Street API,
https://maps.googleapis.com/.

  * •

Population data in Beijing & Shanghai. WorldPop. https://www.worldpop.org/.

  * •

Population & education data in New York. SafeGraph,
https://www.safegraph.com/.

  * •

Crime data in New York. NYC Open Data, https://opendata.cityofnewyork.us/.

Since we focus on the more practical case of irregular region boundaries
partitioned by road network, in the data preprocessing step, we merge multiple
grid-based satellite images to match the region boundary. Thus, the satellite
images for regions might be in irregular shape, as shown in Figure 5(a).

##  Appendix B Model Details & Experiment Results

###  B.1. Training Algorithm

Algorithm 1 summarizes the learning procedure of our proposed KnowCL model for
urban imagery-based socioeconomic prediction. The overall framework is divided
into two steps of knowledge-infused contrastive learning and socioeconomic
prediction. In the first step, lines 4-5 build semantic encoder and visual
encoder for different modalities of inputs, and lines 6-11 execute the cross-
modality based contrastive learning in a minibatch way with model parameter
updated. As for the second step in lines 12-14, only the regression module is
trained with observed socioeconomic indicator data, which is then used for
socioeconomic prediction.

Algorithm 1 Learning procedure of KnowCL model.

1:Input: UrbanKG $\mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{F})$, urban
imagery data $\mathcal{I}$, region set $\mathcal{A}$, socioeconomic indicator
data $\mathcal{D}=\\{(a,y_{a})|a\in\mathcal{A}\\}$.

2:Output: The socioeconomic indicator $y_{a^{\prime}}$ for a region
$a^{\prime}$ without observed label.

3:Step 1: Knowledge-infused Contrastive Learning

4: Initialize semantic encoder $f^{\text{KG}}(\cdot)$;

5: Initialize visual encoder $f^{\text{Image}}(\cdot)$;

6: for $i=1,2,\cdots,n_{\text{iter}}$ do

7: Sample a minibatch $\mathcal{A}_{\text{batch}}\in\mathcal{A}$ of size $m$;

8:
$\bm{e}_{a}=f^{\text{KG}}(\mathcal{G},a),\bm{I}_{a}=f^{\text{Image}}(I_{a}),\forall
a\in\mathcal{A}_{\text{batch}}$;

9:
$\tilde{\bm{e}}_{a}=g^{\text{KG}}(\bm{e}_{a}),\tilde{\bm{I}}_{a}=g^{\text{Image}}(\bm{I}_{a}),\forall
a\in\mathcal{A}_{\text{batch}}$;

10: Compute the image-KG contrastive loss $\mathcal{L}_{a}$ using (5);

11: Update encoder parameters w.r.t. the gradients, $\nabla\mathcal{L}_{a}$.

12:Step 2: Socioeconomic Prediction

13: Train regression module $\text{MLP}(\cdot)$ on $\mathcal{D}$;

14: Predict socioeconomic indicator
$y_{a^{\prime}}=\text{MLP}(f^{\text{Image}}(I_{a^{\prime}}))$.

###  B.2. Predicted v.s. True Indicators

Similar to the setting in Figure 3, we present the comparison of predicted and
true population results on Shanghai and New York in Figure 8. It can be
observed that most predicted samples are located along the line at 45°.

###  B.3. Transferability Study

To validate the transferability of our proposed KnowCL model, we also
investigate the transferability of the best baseline PG-SimCLR in Figure 10.
Compared with results of KnowCL in Figure 6, PG-SimCLR is less competitive in
transfer setting.

###  B.4. Parameter Study

(a) Beijing (b) New York

Figure 11. $R^{2}$ versus the number of street view images per region used for
socioeconomic prediction on two datasets.

Figure 11 further investigates the influence of street view images on Beijing
and New York datasets. Specifically, in the knowledge-infused contrastive
learning step, we keep the number of street view images per region to 10, and
tune the number of street view images used in socioeconomic prediction step.
According to the results, with the increasing of used street view images, the
prediction performance across most of socioeconomic indicators first increases
and then converges. This phenomenon may partly owe to the image quality as
well as the limited information captured in street view imagery. Moreover,
such results also imply that introducing more stree view images for
socioeconomic prediction may not bring performance improvement, which is
heavily affected by the noise therein.

###  B.5. Component Analyses

To analyze the information captured in urban imagery representations, we
employ principal component analysis (PCA) algorithm on learnt visual
representations by KnowCL for dimension reduction, which are presented in
Figure 9. We use the dot color to indicate the population at corresponding
regions. Especially, the clustering phenomenon in respective of populations
can be observed in Figure 9(a)-(d) on Beijing and Shanghai datasets, which
validate the effectiveness of cross-modality based contrastive learning even
without population supervision signals. As for the results in New York
dataset, such phenomenon is not that obvious because the population in New
York is uniformly distributed in block based regions, as we can see that most
dots in Figure 9(e)-(f) are in similar colors. All the experiment results
validate the effectiveness and robustness of our proposed knowledge-infused
contrastive learning model for urban imagery-based socioeconomic prediction.


* West Broida (1975) West1975West, JB. Broida, HP.  1975\.  Chemiluminescence and photoluminescence of diatomic iron oxide Chemiluminescence and photoluminescence of diatomic iron oxide.  The Journal of Chemical Physics6272566-2574.  10.1063/1.430837
  * Wiegert . (2013) Wiegert2013Wiegert, PA., Brown, PG., Weryk, RJ. Wong, DK.  2013\.  The return of the Andromedids meteor shower The return of the Andromedids meteor shower.  The Astronomical Journal145370.  10.1088/0004-6256/145/3/70
  * Yamamoto . (2004) Yamamoto2004Yamamoto, MY., Toda, M., Higa, Y., Maeda, K. Watanabe, JI.  2004\.  Altitudinal distribution of 20 persistent meteor trains: Estimates derived from Metro campaign archives Altitudinal distribution of 20 persistent meteor trains: Estimates derived from Metro campaign archives.  Earth, Moon, and Planets951-4279-287.  10.1007/s11038-005-9048-4
  * Yuan . (2008) Yuan2008Yuan, T., She, CY., Krueger, DA., Sassi, F., Garcia, R., Roble, RG.Schmidt, H.  2008\.  Climatology of mesopause region temperature, zonal wind, and meridional wind over Fort Collins, Colorado (41°N, 105°W), and comparison with model simulations Climatology of mesopause region temperature, zonal wind, and meridional wind over Fort Collins, Colorado (41°N, 105°W), and comparison with model simulations.  Journal of Geophysical Research: Atmospheres113D3D03105.  10.1029/2007JD008697
  * Zhu . (1997) LBFGSBZhu, C., Byrd, RH., Lu, P. Nocedal, J.  1997\.  Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization.  ACM Transactions on Mathematical Software234550–560.  10.1145/279232.279236
  * Zinn Drummond (2005) Zinn2005Zinn, J. Drummond, J.  2005\.  Observations of persistent Leonid meteor trails: 4. Buoyant rise/vortex formation as mechanism for creation of parallel meteor train pairs Observations of persistent Leonid meteor trails: 4. Buoyant rise/vortex formation as mechanism for creation of parallel meteor train pairs.  Journal of Geophysical Research: Space Physics110A4.  10.1029/2004JA010575