# TextRank Based Search Term Identification for Software Change Tasks

Mohammad Masudur Rahman Chanchal K. Roy  University of Saskatchewan, Canada
{masud.rahman<EMAIL_ADDRESS>
###### Abstract

During maintenance, software developers deal with a number of software change
requests. Each of those requests is generally written using natural language
texts, and it involves one or more domain related concepts. A developer needs
to map those concepts to exact source code locations within the project in
order to implement the requested change. This mapping generally starts with a
search within the project that requires one or more suitable search terms.
Studies suggest that the developers often perform poorly in coming up with
good search terms for a change task. In this paper, we propose and evaluate a
novel _TextRank_ -based technique that automatically identifies and suggests
search terms for a software change task by analyzing its task description.
Experiments with 349 change tasks from two subject systems and comparison with
one of the latest and closely related state-of-the-art approaches show that
our technique is highly promising in terms of _suggestion accuracy_ , _mean
average precision_ and _recall_.

###### Index Terms:

Concept location, TextRank, Search Term, Reverse Engineering

##  I Introduction

Studies show that about 80% of the total efforts is spent in software
maintenance and evolution [19]. During maintenance, software developers deal
with a number of software change requests, and identifying the exact location
(i.e., class, method) within a project for a given change task is a major
challenge. Change requests are often made by software users, and are generally
written using natural language texts. The software users might be familiar
with the application domain of a software product; however, they generally
lack the idea of how a particular _software feature_ is implemented in the
source code. Thus a _software change request_ by them mainly involves one or
more domain related concepts, and a developer needs to map those concepts to
the source code locations within the project in order to implement the change
[15, 16]. Such mapping is possibly trivial for a developer who has substantial
knowledge about the target project. However, the developers involved in
maintenance might be unaware of the low-level architecture of the project, and
thus they often experience difficulties in identifying the source locations
(i.e., classes, methods) to be changed. The mapping task generally starts with
a search within the project which requires one or more suitable search terms.
Previous studies [14] suggest that on average, developers with varying
experience perform poorly in coming up with good search terms for a change
task. For example, according to Kevic and Fritz [14], developers can come up
with relevant search terms for only 12.2% of the cases. One way to help them
in this regard is to automatically suggest useful and relevant search terms
for the change task in the first place.

Existing studies that attempt to support developers in _feature location_
tasks with search queries, adopt different lightweight heuristics [14] and
query reformulation or expansion strategies [8, 12, 20], and perform different
query quality analyses [9, 10, 11] and mining activities [15, 13]. However,
most of these approaches expect a developer to provide the initial search
query which they can improve upon, and it is often a non-trivial task for the
developers as noted by other studies too [14]. Kevic and Fritz [14] propose a
heuristic model for automatically identifying initial search terms for a
change task where they consider different heuristics related to _frequency,
location, parts of speech_ and _notation_ of the terms from the task
description. Although the model is found promising in their preliminary
evaluation, it suffers from two major limitations. First, the model is trained
using a limited number of change tasks, and is not cross-validated using the
change tasks from another project. Thus it is still not quite mature or
reliable. Second, _tf–idf_ is one of the dominating metrics in their model,
and it is subject to the size of test dataset for _inverse document frequency
(idf)_ calculation. Thus the same model is likely to perform differently with
different sizes of test dataset, and the model might require frequent re-
training to keep itself useful.

In this paper, we propose a novel _TextRank_ -based technique that
automatically identifies and suggests search terms for software change tasks.
_TextRank_ is an adaptation of _PageRank algorithm_ [7] for natural language
texts where a text document is considered as a lexical or semantic network of
words [18, 6]. _TextRank_ has found its applications in different information
retrieval tasks such as keyword and key phase extraction, extractive
summarization, word sense disambiguation and other tasks involving _graph-
based_ term weighting [18]. Given the successful adoption of that algorithm in
information retrieval domain, it can also be exploited for search term
extraction in the context of _feature location_ tasks. Actually, the algorithm
is highly suited for our purpose from several perspectives. First, software
change requests are generally made by the people outside the development team,
and they communicate their requirements through domain level concepts and
using natural language texts. A _graph-based_ representation (i.e., also
called _text graph_) of the task description can reveal important semantic
(i.e., co-occurrence) relationships among different terms. Second, _TextRank_
algorithm exploits connectivity of a term in the graph, and considers not only
the terms to which the term is connected but also their weights (i.e.,
importance) for determining the weight of that term. The algorithm continues
this process recursively, and thus has a great potential to extract the most
important terms from the graph which can be suggested as search terms.

Issue ID: 401358
Product: JDT, Component: Debug
Summary: Name _selection_ for _Mac_ VM _installs_ needs _improvement_
Description: When you search for a JDK/JRE on _Mac_ , we use information from
the plist file to compute a name. This works fine most of the time, but if you
happen to have more than one of the same version of VM installed they are
added with the same name. To make matters a bit worse, if you edit one of the
_JREs_ the wizard starts out with an error complaining that the name is
already in use. The attached screen shot shows the duplicated names for the
Java 7 _JREs_.

Listing 1: An Example change request from eclipse.jdt.debug

For example, the change task in Listing 1 can be graphically represented as
the _text graph_ in Fig. 1 by analyzing the _co-occurrence relationships_
among the words with a _window size_ (i.e., number of words considered in a
semantic text unit) of two. Our proposed technique analyzes the term
connectivity, calculates weight (i.e., importance) for each of the terms in
the graph, and then extracts the top-scored and the most suitable five terms–
_Mac, selection, installs, improvement_ and _JREs_ as the initial search query
for the task.

Experiments with 349 change tasks from two subject systems– _Apache Log4j_ and
_Eclipse JDT Debug_ show that our approach can return relevant results (i.e.,
Java classes) for 49.34% of the change tasks with 57.16% _mean average
precision_ and 63.33% _recall_. We also compare with one of the latest and
closely related state-of-the-art approaches– Kevic and Fritz [14], and report
that our approach performs relatively better in terms of all performance
metrics. While our preliminary results are found promising, they must be
further validated with more change tasks from more subject systems. Thus the
paper makes the following technical contributions:

  * •

It demonstrates a novel use of _TextRank_ algorithm in the identification and
suggestion of relevant search terms for software change tasks.

  * •

It reports a case study involving 349 software change tasks from two subject
systems and comparison with one existing approach [14], and shows the
effectiveness of the proposed technique.

The rest of the paper is organized as follows – Section II explains our
adopted methodology and algorithms, Section III discusses the conducted
experiments and findings, Section IV describes the related work, and finally
Section V concludes the paper with future work.

Figure 1: Text Graph of change request in Listing 1 Figure 2: Schematic
diagram of the proposed technique

##  II Proposed Methodology

Fig. 2 shows the schematic diagram of our proposed technique for search term
identification and suggestion for a change task. In this section, we discuss
the different steps involved with the technique as follows:

###  II-A Data Collection

In order to suggest search terms for a change task, we make use of natural
language description (e.g., Listing 1) for the task provided by the user. We
collect actual task descriptions of 349 tasks from BugZilla official
repositories. Each of those tasks is submitted using a semi-structured way,
and they contain several fields such as _Issue ID_ (e.g., 401358), _Product_
(e.g., JDT), _Component_ (e.g., Debug), _Summary_ and _Description_. We use
the last two fields for the analysis by our technique. While _Summary_ shows
the title of a requested change task, _Description_ contains the user’s
explanation of the task in natural language texts. In order to keep the
algorithm simple and lightweight, we do not consider content from the files
attached to the change request. Adding more available information about the
task is likely to improve the performance of our technique; however, we
consider that part as a scope of future study.

###  II-B Text Preprocessing

We analyze _Summary_ and _Description_ of a software change request, and
perform several preprocessing before transforming the texts into a _text
graph_. We consider each sentence as a _logical text unit_ that contributes to
the overall task description, and collect each of them from those fields. We
attempt to extract such terms from a sentence that either convey useful
semantics or represent the domain level concepts of the software to be
changed. We thus remove stop words (i.e., frequently used non-important words)
from each of those sentences, and split the dotted structured words (i.e.,
containing dots) into simpler words from them. A dotted word often involves
multiple technical concepts, and splitting helps one to analyze each of them
in isolation. For example the word– org.eclipse.ui.part.PageBookView.create
PartControl contains a _package name_ (i.e.,  _org.eclipse.ui.part_), a _class
name_ (i.e.,  _PageBookView_) and a _method name_ (i.e.,
_createPartControl_), and only splitting helps to analyze those concepts
effectively. It should be noted that we do not split the words based on camel-
case notation given these two observations. First, change requests often
contain different technical artifacts such as _library name_ , _class name_
and _method name_ using camel-case notations in the description, and they are
of great interest to the developers implementing those changes. Splitting such
words simply breaks those artifacts, and does not help in identifying suitable
search terms. Second, experiments with such splitting do not result into
performance improvement of our technique. We also avoid stemming on the words
of the task description as it does not demonstrate any notable improvement in
the performance of our technique.

###  II-C Text Graph Development

After preprocessing, we get a list of sentences each of which contains
semantically important and domain related terms, and we use them to develop a
_term graph_ for the task description. We represent each of those terms as a
distinct node in the graph, and consider the _co-occurrence_ of those terms in
the sentences as an indication of semantic relationships (i.e., dependencies)
among them [18, 6]. For example, if we consider the sentence– _”This works
fine most of the time, but if you happen to have more than one of the same
version of VM installed they are added with the same name”_ from the example
description (Listing 1), the preprocessed version forms an ordered list of
terms– _”works fine time happen version installed”_. We note that the
transformed sentence contains several phrases such as _works fine_ and
_version installed_ , and the terms in those phrases are semantically
dependent on each other for comprehensive meaning. We thus consider a _window
size_ of two (i.e., best performing size according to Mihalcea and Tarau [18])
as a semantic unit of words, and get the following relationships– _works_
$\longleftrightarrow$_fine_ , _fine_ $\longleftrightarrow$_time_ , _time_
$\longleftrightarrow$_happen_ , _happen_ $\longleftrightarrow$_version_ , and
_version_ $\longleftrightarrow$_installed_. We then encode such relationships
into the connecting edges among the corresponding nodes in the graph.

###  II-D TextRank Calculation

In order to estimate the weight (i.e., importance) of each of the terms in
_text graph_ , we use _TextRank_ algorithm by Mihalcea and Tarau [18] which is
adapted from the popular _PageRank_ algorithm [7] for web link analysis.
_TextRank_ analyzes the connectivity details such as incoming links and
outgoing links of each term in the graph recursively, and calculates its
weight, $S(v_{i})$, as follows:

$S(v_{i})=(1-\phi)+\phi\sum_{j\epsilon
In(v_{i})}\frac{S(v_{j})}{|Out(v_{j})|}~{}~{}(0\leq\phi\leq 1)$

Here, $In(v_{i})$, $Out(v_{j})$, and $\phi$ denote the node list connected to
$v_{i}$ through incoming links, the node list to which $v_{j}$ is connected
through outgoing links, and the dumping factor respectively. In the _text
graph_ , each of the edges is bi-directional (i.e., terms depend on each
other), and thus _in-degree_ is equal to the _out-degree_ for a node (i.e.,
term). In the context of web surfing, dumping factor, $\phi$, is considered as
the probability of randomly choosing a web page by the surfer, and $1-\phi$ is
the probability of jumping off that page. Mihalcea and Tarau [18] use a
heuristic value of 0.85 for $\phi$, and we also use the same value for
_TextRank_ calculation. We initialize each of the terms in the graph with a
default value of 0.25, and run an iterative version of the algorithm [7]. It
should be noted that the initial value of a term does not affect its final
score [18]. The computation iterates until the scores of the terms converge
below a certain threshold or it reaches the maximum iteration limit (i.e., 100
as suggested by Blanco and Lioma [6]). As Mihalcea and Tarau [18] suggest, we
use a heuristic threshold of 0.0001 for the convergence checking. _TextRank_
applies the underlying mechanism of a recommendation (i.e., voting) system,
where a term recommends (i.e., votes) another term if the second term
complements the semantics of the first term in any way [18]. The algorithm
captures recommendation for a term in terms of incoming links in the _text
graph_ (e.g., Fig. 1) from another terms both in local (i.e., same sentence)
and global (i.e., entire document) context, and determines importance of the
term. Once computation is over, each of the terms in the graph is found with a
score which is considered as the weight or importance of that term in the
texts.

###  II-E Search Term Selection

Once _TextRank_ is calculated, we rank the terms based on their weights (i.e.,
importance), and choose the search terms for a change task in a heuristic
fashion. According to Kevic and Fritz [14], the terms that exist both in
_Summary_ and _Description_ of a task are the most suitable for search terms.
We adapt that idea given the fact that the overlapping terms in those fields
might not be sufficient enough to form a search query. We thus first look for
top-scored five terms in the _Summary_ (i.e., title) of a change task. If the
_Summary_ is too small to provide all the terms, we collect the rest from the
_Description_ of the task. It should be noted that in both cases, terms are
chosen based on their ranks or weights which are calculated by recursively
analyzing the surrounding terms in the text graph. For instance, in the case
of our example change task (Listing 1), the first four search terms– _Mac
(score: 0.64), selection (score: 0.27), installs (score: 0.27)_ and
_improvement (score: 0.25)_ come from the title. The remaining search term–
_JREs (score: 1.00)_ , one of the most important terms, is not contained in
the title, and thus it comes from the description.

TABLE I: Experimental Results

Metric | Log4j | eclipse.jdt.debug | Average
---|---|---|---
$\mathbf{T_{3}}{\textsuperscript{1}}$ | $\mathbf{T_{4}}{\textsuperscript{2}}$ |  $\mathbf{T_{5}}$3 | $\mathbf{T_{3}}$ | $\mathbf{T_{4}}$ | $\mathbf{T_{5}}$ | $\mathbf{T_{5}}$
No. of Tasks Solved (NTS) | 86(230) | 98(230) |  111(230) | 48(119) | 54(119) |  60(119) | –
% of Tasks Solved (PTS) | 37.39% | 42.61% |  48.26% | 40.34% | 45.38% |  50.42% | 49.34%
Mean Average Precision (MAP) |  66.54% | 62.82% | 61.16% |  56.79% | 51.45% | 53.16% | 57.16%
Mean Recall (MR) | 53.22% |  55.54% | 54.66% |  73.56% | 73.27% | 72.00% | 63.33%

1Results for three search terms, 2Results for four search terms, 3Results for
five search terms

##  III Experiment

In order to evaluate our proposed technique, we conduct experiments using the
change tasks from two subject systems. We also compare with one existing
approach, and this section discusses our evaluation and validation details.

###  III-A Dataset

In our experiments, we use 349 change tasks from two subject systems– Log4j by
_Apache_ and eclipse.jdt.debug by _Eclipse_. Each of those tasks were marked
as _RESOLVED_ , and we follow a careful approach for their selection. First we
collect all the _RESOLVED_ change tasks from the BugZilla repositories [3, 1],
and then attempt to map them against the commit history of the corresponding
projects (i.e., repositories) at GitHub [4, 2]. We analyze the commit messages
of each project, and look for specific _Bug IDs_ (i.e., identifiers of change
tasks) in those messages. In GitHub, we note that each commit operation that
solves a software bug or addresses a change request, generally mentions the
corresponding _Bug ID_ in the very first sentence of its commit message. We
found such 230 commits in Log4j and 119 commits in eclipse.jdt.debug. We then
collect the _changeset_ (i.e., list of changed files) for each of those commit
operations, and develop a solution set for the corresponding change tasks.
Thus, for our experiments, we collect not only the actual change tasks from
the reputed subject systems but also their solutions that were applied in
practice by the developers. We use different utility commands such as _git,
clone, rev-list_ and _log_ on _GitHub Bash_ for collecting those information.

###  III-B Search Engine

We use a _vector space model_ based search engine– _Apache Lucene_ [19] for
searching the files that need to be changed for a change task. The search
engine indexes the files in the corpus prior to search, and we note that the
indexing by _Lucene Indexer_ is not efficient. Especially the terms indexed
from the source files are not meaningful and they often contain different
special characters (i.e., punctuations). The indexer is basically targeted for
simple text documents whereas the source files in the project contain items
beyond regular texts (e.g., code segments). We thus apply limited
preprocessing (i.e., stemming was avoided) on each of those source files in
each project, and remove all punctuation characters from them. This transforms
the source files into text like files, and the indexer becomes able to perform
more effectively, especially in choosing meaningful index terms. Once a search
is initiated using a query, the search engine filters irrelevant files in the
corpus using a _boolean search model_ , and then applies a _tf-idf_ based
scoring technique to return a ranked list of relevant documents. As existing
studies suggest [5, 14, 17], we consider only the top ten results from the
search engine for performance evaluation and validation of our technique.

TABLE II: Comparison with an Existing Approach

Technique | Metric | Log4j | eclipse.jdt.debug
---|---|---|---
$\mathbf{T_{3}}{\textsuperscript{1}}$ |  $\mathbf{T_{5}}$2 | $\mathbf{T_{3}}$ | $\mathbf{T_{5}}$
Kevic and Fritz [14] | No. of Tasks Solved (NTS) | 47(230) |  65(230) | 27(119) |  39(119)
% of Tasks Solved (PTS) | 20.43% | 28.26% | 22.69% | 32.77%
Mean Average Precision (MAP) | 54.08% |  56.90% | 50.61% |  54.92%
Mean Recall (MR) |  50.39% | 48.36% | 66.70% |  78.53%
Proposed | No. of Tasks Solved (NTS) | 86(230) |  111(230) | 48(119) |  60(119)
% of Tasks Solved (PTS) | 37.39% | 48.26% | 40.34% | 50.42%
Mean Average Precision (MAP) |  66.54% | 61.16% |  56.79% | 53.16%
Mean Recall (MR) | 53.22% |  54.66% |  73.56% | 72.00%

1Results for three search terms, 2Results for five search terms

###  III-C Performance Metrics

Mean Average Precision at K (MAPK): _Precision at K_ calculates _precision_ at
the occurrence of every relevant result in the ranked list. _Average Precision
at K (APK)_ averages the _precision at K_ for all relevant results in the list
for a search query. Thus _Mean Average Precision at K (MAPK)_ is calculated
from the mean of _average precision at K_ for all queries in the dataset as
follows:

$APK=\frac{\sum_{k=1}^{D}P_{k}\times
rel_{k}}{\left|S\right|},~{}~{}MAPK=\frac{\sum_{q\epsilon
Q}APK(q)}{\left|Q\right|}$

Here, $rel_{k}$ denotes the relevance function of $k^{th}$ result in the
ranked list, $P_{k}$ denotes the precision at $k^{th}$ result, and $D$ refers
to number of total results. $S$ is the solution set for a query, and $Q$ is
the set of all queries.

Mean Recall (MR): _Recall_ denotes the fraction of the solution set that is
retrieved for a search query. _Mean Recall_ averages such measures for all
queries in the dataset.

###  III-D Experimental Results

We conduct experiments using 349 change tasks from two subject systems, and
apply four different metrics– _no. of tasks solved_ , _% of tasks solved_ ,
_mean average precision_ and _mean recall_ for performance evaluation. We
consider different sizes for the search query, and collect the performance
details of our suggested queries for both subject systems which are reported
in Table I. From the table, we note that our queries perform the best when
five terms are used for search. For example, they return relevant results for
111 (48.26%) out of 230 change tasks from Log4j, and for 60 (50.42%) out of
119 change tasks from eclipse.jdt.debug, which is promising. Thus, on average,
our queries retrieve 63.33% of the solutions from the dataset with a _mean
average precision_ of 57.16% for each of the subject systems. We also conduct
search using six query terms; however, we note that our queries perform
relatively poor in that regard for both subject systems.

We investigate whether the _preprocessing_ (Section III-B) of corpus files
significantly influences the performance of our queries. We re-ran the
experiments on the corpus without preprocessing, and did not experience any
major performance degradation. Thus, the findings actually demonstrate the
robustness of our suggested search terms for a change task. We also
investigate whether a list of randomly chosen five search terms from the
_Summary_ of change task is comparable to our suggested search terms given
that our algorithm also emphasizes on _Summary_ terms (Section II-E). We
conducted experiments with such queries from both subject systems where we
noted that the queries performed quite poorly. For example, the search engine
returns relevant results for at most 54 (compared to 111) tasks from Log4j and
39 (compared to 60) tasks from eclipse.jdt.debug. Thus, the findings
demonstrate that our proposed technique for search term suggestion is
relatively more effective and more reliable.

###  III-E Comparison with an Existing Approach

We compare with one of the latest and closely related state-of-the-art
approaches– Kevic and Fritz [14] using our dataset (Section III-A). Kevic and
Fritz [14] propose a heuristic model for search term suggestion for a change
task where they consider frequency (i.e.,  _tf-idf_), location (i.e.,
_inSumAndBody, isInMiddle_) and notation (i.e.,  _isCamelCase_) of the terms
from the task description. They suggest three search terms as a search query
whereas we find our technique performing the best with five search terms in a
query. We thus consider both sizes for the queries in our experiments. We
implement their _relevance model_ in our working environment, collect the
search queries for the change tasks, and evaluate them using the same search
engine (Section III-B). From Table II, we note that our queries are more
effective compared to theirs for both subject systems. For example, their
queries can retrieve relevant results for at most 65 (28.26%) out of 230
change tasks from Log4j and 39 (32.77%) out of 119 tasks from
eclipse.jdt.debug. In terms of _precision_ and _recall_ , we note that our
queries also perform relatively better than theirs.

##  IV Related Work

There exist a number of studies in the literature that attempt to support
developers in _feature location_ tasks with search queries. They adopt
different lightweight heuristics [14] and query reformulation or expansion
strategies [8, 12, 20], and perform different query quality analyses [9, 10,
11] and mining activities [15, 13]. However, most of these approaches expect a
developer to provide the initial search query which they can improve upon, and
their main focus is on improving a given query for a change task. On the other
hand, in this study, we propose a novel technique that suggests a list of
suitable terms as an initial search query for the given task. From technical
perspective, we adapt an established algorithm– _TextRank_ from information
retrieval domain that analyzes the relative importance of the terms from the
task description using a graph-based technique, and suggests the most
important terms as search terms. Besides, we perform a case study using 349
change tasks that demonstrates the potential of the adapted technique.

##  V Conclusion & Future Work

To summarize, in this paper, we propose a novel _TextRank_ -based technique
that automatically identifies and suggests search terms for software change
tasks. Experiments with 349 change tasks from two subject systems show that
our approach, on average, can return relevant results (i.e., Java classes) for
49.34% of the change tasks with 57.16% _mean average precision_ and 63.33%
_recall_. Comparison with one of the latest and closely related state-of-the-
art approaches also shows that our approach performs comparatively better in
different performance metrics. While these preliminary findings demonstrate
the high potential of the proposed approach, further validation with more
subject systems of diverse varieties and change tasks is warranted.

## References

  * deb [a] JDT, a.  URL https://bugs.eclipse.org/bugs/describecomponents.cgi?product=JDT.
  * deb [b] JDT Repository, b.  URL https://github.com/eclipse/eclipse.jdt.debug.
  * log [a] a.  URL https://issues.apache.org/bugzilla/describecomponents.cgi?product=Log4j.
  * log [b] Log4j Repository, b.  URL https://github.com/apache/log4j.
  * Antoniol et al. [2002] G. Antoniol, G. Canfora, G. Casazza, A. De Lucia, and E. Merlo.  Recovering Traceability Links between Code and Documentation.  _TSE_ , 28(10):970–983, 2002.
  * Blanco and Lioma [2012] R. Blanco and C. Lioma.  Graph-based Term Weighting for Information Retrieval.  _Inf. Retr._ , 15(1):54–92, 2012.
  * Brin and Page [1998] S. Brin and L. Page.  The Anatomy of a Large-Scale Hypertextual Web Search Engine.  _Comput. Netw. ISDN Syst._ , 30(1-7):107–117, 1998.
  * Gay et al. [2009] G. Gay, S. Haiduc, A. Marcus, and T. Menzies.  On the Use of Relevance Feedback in IR-based Concept Location.  In _Proc. ICSM_ , pages 351–360, 2009.
  * Haiduc [2011] S. Haiduc.  Automatically Detecting the Quality of the Query and its Implications in IR-based Concept Location.  In _Proc. ASE_ , pages 637–640, 2011.
  * Haiduc and Marcus [2011] S. Haiduc and A. Marcus.  On the effect of the query in ir-based concept location.  In _Proc. ICPC_ , pages 234–237, June 2011.
  * Haiduc et al. [2012] S. Haiduc, G. Bavota, R. Oliveto, A. Marcus, and A. De Lucia.  Evaluating the Specificity of Text Retrieval Queries to Support Software Engineering Tasks.  In _Proc. ICSE_ , pages 1273–1276, 2012.
  * Haiduc et al. [2013] S. Haiduc, G. Bavota, A. Marcus, R. Oliveto, A. De Lucia, and T. Menzies.  Automatic Query Reformulations for Text Retrieval in Software Engineering.  In _Proc. ICSE_ , pages 842–851, 2013.
  * Howard et al. [2013] M.J. Howard, S. Gupta, L. Pollock, and K. Vijay-Shanker.  Automatically Mining Software-based, Semantically-Similar Words from Comment-Code Mappings.  In _Proc. MSR_ , pages 377–386, 2013.
  * Kevic and Fritz [2014a] K. Kevic and T. Fritz.  Automatic Search Term Identification for Change Tasks.  In _Proc. ICSE_ , pages 468–471, 2014a.
  * Kevic and Fritz [2014b] K. Kevic and T. Fritz.  A Dictionary to Translate Change Tasks to Source Code.  In _Proc. MSR_ , pages 320–323, 2014b.
  * Marcus and Haiduc [2013] A. Marcus and S. Haiduc.  Text Retrieval Approaches for Concept Location in Source Code.  In _Software Engineering_ , volume 7171, pages 126–158. 2013.
  * Marcus and Maletic [2003] A. Marcus and J.I. Maletic.  Recovering Documentation-to-Source-Code Traceability Links Using Latent Semantic Indexing.  In _Proc. ICSE_ , pages 125–135, 2003.
  * Mihalcea and Tarau [2004] R. Mihalcea and P. Tarau.  Textrank: Bringing Order into Texts.  In _Proc. EMNLP_ , pages 404–411, 2004.
  * Ponzanelli et al. [2013] L. Ponzanelli, A. Bacchelli, and M. Lanza.  Seahawk: Stack Overflow in the IDE.  In _Proc. ICSE_ , pages 1295–1298, 2013.
  * Shepherd et al. [2007] D. Shepherd, Z. P. Fry, E. Hill, L. Pollock, and K. Vijay-Shanker.  Using Natural Language Program Analysis to Locate and Understand Action-Oriented Concerns.  In _Proc. ASOD_ , pages 212–224, 2007.


# Sat2Density: Faithful Density Learning from Satellite-Ground Image Pairs

Ming Qian1,2 Jincheng Xiong1 Gui-Song Xia1 Nan Xue2
1LIESMARS Corresponding author  Wuhan University 2Ant Group
https://sat2density.github.io

###### Abstract

This paper aims to develop an accurate 3D geometry representation of satellite
images using satellite-ground image pairs. Our focus is on the challenging
problem of 3D-aware ground-views synthesis from a satellite image. We draw
inspiration from the density field representation used in volumetric neural
rendering and propose a new approach, called Sat2Density. Our method utilizes
the properties of ground-view panoramas for the sky and non-sky regions to
learn faithful density fields of 3D scenes in a geometric perspective. Unlike
other methods that require extra depth information during training, our
Sat2Density can automatically learn accurate and faithful 3D geometry via
density representation without depth supervision. This advancement
significantly improves the ground-view panorama synthesis task. Additionally,
our study provides a new geometric perspective to understand the relationship
between satellite and ground-view images in 3D space.

##  1 Introduction

The emergence of satellite imagery has significantly enhanced our daily lives
by providing easy access to a comprehensive view of the planet. This
bird’s-eye view offers valuable information that compensates for the limited
perspective of ground-level observations by humans. However, what specific
information does satellite imagery provide, and why is it so crucial? In this
paper, we propose that the most critical insights come from the analysis of
the geometry, topology, and geography of cross-view observations captured by
paired satellite and ground-level images. Building on this hypothesis, we aim
to address the challenging problem of synthesizing ground-level images from
paired satellite and ground-level imagery by leveraging density
representations of 3D scenes.

(a) Learned density from satellite images

(b) Synthesized panoramas

(c) Rendered depth

Figure 1: Sat2Density trains with a collection of satellite-ground image
pairs, without depth, or multi-view supervision. Our 3d GAN enables the
synthesis of scenes conditioned on a satellite image, producing multi-view-
consistent ground-view renderings and geometry. Please see the project page
for more videos.

The challenge of generating ground-level images from satellite imagery is
tackled by leveraging massive datasets containing both satellite images and
corresponding ground-level panoramas captured at the same geographical
coordinates. However, the drastic differences in viewpoint between the two
types of images, combined with the limited overlap of visual features and
large appearance variations, create a highly complex and ill-posed learning
problem. To address this challenge, researchers have extensively studied the
use of conditional generative adversarial networks, which leverage high-level
semantics and contextual information in a generative way [19, 20, 33, 26, 12].
However, since the contextual information used is typically at the image
level, the 3D information can only be marginally inferred during training,
often resulting in unsatisfactory synthesis results.

Recent studies [22, 11] have suggested that accurate 3D scene geometry plays a
crucial role in generating high-quality ground-view images. With extra depth
supervision, Sat2Video [11] introduced a method to synthesize spatial-temporal
ground-view video frames along a camera trajectory, rather than a single
panorama from the center viewpoint of the satellite image. Additionally, Shi
_et al_. [22] demonstrated that coarse satellite depth maps can be learned
from paired data through multi-plane image representation using a novel
projection model between the satellite and ground viewpoints, but a coarse 3d
representation can not facilitate rendering 3D-aware ground-view images.
Building on these insights, we aim to investigate whether it is possible to
achieve even more accurate 3D geometry using the vast collection of satellite-
ground image pairs.

Our study is motivated by the latest developments in the neural radiance field
(NeRF) [16], which has shown promising results in novel view synthesis.
Benefiting from the flexibility of density field in volumetric rendering [8],
faithful 3D geometry can be learned from a large number of posed images.
Therefore, we adopt density fields as the representation and focus on learning
accurate density fields from paired satellite-ground image pairs. More
precisely, in this paper, we present a novel approach called Sat2Density,
which involves two convolutional encode-decoder networks: DensityNet and
RenderNet. The DensityNet receives satellite images as input to represent the
density field in an explicit grid, which plays a crucial role in producing
ground-view panorama images using the RenderNet. With such a straightforward
network design, we delve into the goal of learning faithful density field
first and then render high-fidelity ground-view panoramas.

While we employed a flexible approach to representing geometry using explicit
volume density and volumetric rendering, an end-to-end learning approach alone
is inadequate for restoring geometry using only satellite-ground image pairs.
Upon examining the tasks and satellite-ground image pairs, we identified two
main factors that may impede geometry learning, which has been overlooked in
previous works on satellite-to-ground view synthesis. Firstly, the sky is an
essential part of ground scenes but is absent in the satellite view, and it is
nearly impossible to learn a faithful representation of the infinite sky
region in each image using explicit volume density. Secondly, differences in
illumination among the ground images during training make it challenging to
learn geometry effectively.

With the above intuitive observation, we propose two supervision signals, the
_non-sky opacity supervision_ and _illumination injection_ , to learn the
density fields in a volumetric rendering form jointly. The _non-sky opacity
supervision_ compels the density field to focus on the satellite scene and
ignore the infinity regions, whereas the _illumination injection_ learns the
illumination from sky regions to further regularize the learning density
field. By learning the density field, our Sat2Density approach goes beyond the
center ground-view panorama synthesis from the training data and achieves the
ground-view panorama video synthesis with the best spatial-temporal
consistency. As shown in Figure 1, our Sat2Density continuously synthesizes
the panorama images along the camera trajectory. We evaluated the
effectiveness of our proposed approach on two large-scale benchmarks [22, 34]
and obtained state-of-the-art performance. Comprehensive ablation studies
further justified our design choices.

The main contributions of our paper are:

  * •

We present a geometric approach, Sat2Density, for ground-view panorama
synthesis from satellite images in end-to-end learning. By explicitly modeling
the challenging cross-view synthesis task in the density field for the 3D
scene geometry, our Sat2Density is able to synthesize high-fidelity panoramas
on camera trajectories for video synthesis without using any extra 3D
information out of the training data.

  * •

We tackle the challenging problem of learning high-quality 3D geometry under
extremely large viewpoint changes. By analyzing the unique challenges that
arise with this problem, we present two intuitive approaches _non-sky opacity
supervision_ and _illumination injection_ to compel the density learning to
focus on the relevant features in the satellite scene presented in the paired
data while mitigating the effects of infinite regions and illumination
changes.

  * •

To the best of our knowledge, we are the first to successfully learn a
faithful geometry representation from satellite-ground image pairs. We believe
that not only do our new findings improve the performance of ground-view
panorama synthesis, but the learned faithful density will also provide a
renewed understanding of the relationship between satellite and ground-view
image data from a 3D geometric perspective.

Figure 2: Overview of Sat2Density. The generation consists of two components,
DensityNet and RenderNet. We optimize Sat2Density by reconstruction loss,
adversarial loss, illumination injection loss, and opacity loss. See text for
details.

##  2 Related Work

###  2.1 Satellite-Ground Cross-View Perception

Both ground-level and satellite images provide unique perspectives of the
world, and their combination provided us with a more comprehensive way to
understand and perceive the world from satellite-ground visual data. However,
due to the drastic viewpoint changes between the satellite and ground images,
poses several challenges in geo-localization [34, 23, 24, 25], cross-view
synthesis [22, 11, 12, 20, 26], overhead image segmentation with the
assistance of ground-level images [30], geo-enabled depth estimation [29],
predicting ground-level scene layout from aerial imagery [33].

To address this challenge, many previous works have proposed various
approaches to model and learn the drastic viewpoint changes, including the use
of homography transforms [20], additional depth or semantic supervision [26,
12, 11], transformation matrices [34], and geospatial attention [30], among
others. Despite effectiveness, these approaches mainly address the challenge
on the image level instead of the 3D scenes.

Most recently, Shi _et al_. [22] proposed a method to learn geometry in
satellite scenes implicitly using the height (or depth) probability
distribution map, which achieved better results in synthesized road and
grassland regions through their geometry projection approach. However, their
learned geometry has limited effectiveness as the rendered satellite depth
cannot accurately recognize objects. We go further along the line to focus on
the 3D scene geometry conveyed in the satellite-ground image pairs. We
demonstrate that the faithful 3D scene geometry can be explicitly decoded and
leveraged with an appropriate representation and supervision signals, to
obtain high-fidelity ground-view panoramas. Besides, we believe that our study
brings a novel perspective to rethink satellite-ground image data for many
other challenging problems.

###  2.2 Neural Radiance Field

Benefiting from the flexibility of density field in volumetric rendering [16],
faithful 3D geometry can be learned from a dense number of posed images [4,
13, 1, 3]. Recent works [2, 31, 32] based on NeRF have shown that 3D
representation can be learned even with only a few views. In a co-current work
[28], it is also pointed out that the flexibility of the density field helps
to learn the 3D geometric structure from a single image by disentangling the
color and geometry, which allows neural networks to capture reliable 3D
geometry in occluded areas.

Our goal can be viewed as an extremely challenging problem of density-based
few-view synthesis with extremely large viewpoint changes, which was not
studied well in previous works. In our study, we demonstrated the possibility
of learning faithful geometry in the volumetric rendering formulation,
shedding light on the most challenging cross-view configurations for novel
view synthesis.

##  3 The Proposed Sat2Density

Figure 2 illustrates the computation pipeline for our proposed Sat2Density.
Given the input satellite image $I_{\text{sat}}\in\mathbb{R}^{H\times W\times
3}$ for the encoder-decoder DensityNet, we learn an explicit volume of the
density field $V_{\sigma}\in\mathbb{R}^{H\times W\times N}$. We render the
panorama depth and project the color of the satellite image along rays in the
ground view to generate an initial panorama image and feed them to the
RenderNet. To ensure consistent illumination of the synthesis, the histogram
of color in the sky regions of the panorama is used as a conditional input for
our method.

###  3.1 Density Field Representation

We encode an explicit volume density as a discrete representation of scene
geometry and parameterize it using a plain encoder-decoder architecture in
DensityNet $G_{\text{dns}}$ to learn the density field:

$V_{\sigma}=G_{\text{dns}}(I_{\text{sat}})\ \ \ \ \ \text{s.t.}\
V_{\cdot,\cdot,\cdot}\in[0,\infty).$ (1)

where the density information $v=V(x,y,z)$ is stored in the volume of $V$ for
the spatial location $(x,y,z)$. For any queried location that does not locate
in the sample position of the explicit grid, tri-linear interpolation is used
to obtain its density value. Suppose the size of the real-world cube is
$(X,Y,Z)$ in the satellite image, two corner cases are considered: 1) for the
locations outside the cube, we set their density to zero, and 2) we set the
density in the lowest volume (_i.e_., V(x,y,z=0)) to a relatively large value
($10^{3}$ in our experiments), which made an assumption that all ground
regions are solid.

With the density field representation, the volumetric rendering techniques
[14] are applied to render the depth $\hat{d}$ and opacity $\hat{O}$ along the
queried rays by

$\hat{d}=\sum_{i=1}^{S}T_{i}\alpha_{i}d_{i},\qquad\hat{O}=\sum_{i=1}^{S}T_{i}\alpha_{i},$
(2)

where $d_{i}$ is the distance between the camera location and the sampled
position, $T_{i}$ denotes the accumulated transmittance along the ray at
$t_{i}$, and $\alpha_{i}$ is the alpha value for the alpha compositing,
written by

$\alpha_{i}=1-\exp\left(-\sigma(\mathbf{x}_{i})\delta_{i}\right)\quad
T_{i}=\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right).$ (3)

Unlike NeRF [16] that learns the radiance field to render the colored images,
we take a copy-paste strategy to compute the colored images by copying the
color from the satellite image along the ray via bilinear interpolation for
image rendering in

$\hat{c}_{\rm map}=\sum_{i}^{S}T_{i}\alpha_{i}c_{i},$ (4)

where $c_{i}=c(x_{i},y_{i},z_{i})=I_{\rm
sat}(\frac{x_{i}}{S_{x}}+\frac{H}{2},\frac{y_{i}}{S_{y}}+\frac{W}{2})$.
$S_{x}$ and $S_{y}$ are the scaling factors between the pixel coordinate of
the satellite image and the grid coordinate in $V_{\sigma}$. To keep the
simplicity of our Sat2Density, we did not use the hierarchical sampling along
rays for the computation of depth, colors, and opacity.

Thanks to the flexibility of volumetric rendering, for the end task of ground-
view panorama synthesis, it is straightforward to render the ground-view
depth, opacity, and the (initial) colored image. For the subsequent RenderNet,
it takes the concatenated tensor of the rendered panorama depth, opacity, and
colors as input to synthesize the high-fidelity ground-view images.

Learning density could draw precise geometry information of the scene, but it
is hard to acquire real density information of the satellite scene only from
the satellite-ground image pairs. In our work, we propose two supervisions:
non-sky opacity supervision and illumination injection to improve the quality
of the 3D geometry representation.

###  3.2 Supervisions from Sky/Non-Sky Separation

#### Non-Sky Opacity Supervision.

We draw inspiration from the study of panorama image segmentation [15, 35],
which treats the sky region as a meaningful category in the segmentation task.
By taking the off-the-shelf sky segmentation model [35] to obtain the sky
masks for the training panorama images, we tackle the infinity issue with a
novel non-sky opacity supervision proposed. Based on our discussion in Sec. 1,
the pseudo sky masks provide a strong inductive basis to faithfully learn the
density fields for our proposed Sat2Density in a simple way.

Denoted by $\mathcal{R}$ and $\mathcal{R}^{\prime}$ the non-sky/sky regions of
the ground-view panorama, the loss function $\mathcal{L}_{\rm snop}$ of our
proposed non-sky opacity supervision reads to

$\mathcal{L}_{\rm
snop}=\sum_{\mathbf{r}\in\mathcal{R}}\left\|\hat{O}(\mathbf{r})-1\right\|_{1}+\sum_{\mathbf{r}\in\mathcal{R}^{\prime}}\left\|\hat{O}(\mathbf{r})\right\|_{1}.$
(5)

#### Illumination Injection from Sky Regions.

While the density field works well on images of static subjects captured under
controlled settings, it is incapable of modeling many ubiquitous, real-world
phenomena for ground-view panorama synthesis. More importantly, due to the
lacking of correspondence from the sky regions in ground-view images to the
paired satellite image, we find that the variable illumination in the ground
images is a key factor preventing the model to learn faithful 3D geometry.

Accordingly, we present an illumination injection from sky regions of the
panorama. For the sake of simplicity of design, we choose the RGB histogram
information in the sky regions as the illumination hints. In our
implementation, we first cut out the sky part from the ground image, then
calculate the RGB sky histogram with $N$ bins. To further exploit the
representational ability of sky histograms, we follow the style encoding
scheme proposed in SPADE [17] to transform the sky histogram into a fixed-dim
embedding, which allows our Sat2Density learn reliable information of
complicated illumination from the sky histogram for the RenderNet. From our
experiments, we also find that the injection of sky histogram into the
RenderNet could further improve the quality of explicit volume density by
encouraging the DensityNet to focus on the stuff regions rather than being
disturbed by the per-image illumination variations issue.

By combining the above two approaches, we solve the per-image illumination
variations and infinity issue, and let our model focus on learning the scene
geometry relationship between satellite and ground view (see Figure 3).
Besides, thanks to the proposed sky histogram illumination injection approach,
our model could control illuminations.

###  3.3 Loss Functions

Sat2Density is trained with both reconstruction loss and adversarial loss. For
reconstruction loss, we follow GAN-based syntheses works, using a combination
of the perceptual loss [7], L1, and L2 loss. In the adversarial loss, we use
the non-saturated loss [9] as the training objective. Besides,
$\mathcal{L}_{snop}$ is used for opacity supervision. For illumination
learning, we follow the SPADE [18] use a KL Divergence loss. Last, in the
discriminator, we use the feature matching loss & a modified multi-scale
discriminator architecture in [27]. Details can be found in the supplemental
material.

##  4 Experiments

###  4.1 Implementation Details

We train our model with $256\times 256$ input satellite images and output a
$256\times 256\times 65$ explicit volume density and finally predict a
$128\times 512$ 360° panorama image, which is the same as the setting in [22]
for a fair comparison. The maximum height modeled by our implicit volume
density is 8 meters, which is an empirical setup. We approximate the height of
the street-view camera as 2 meters with respect to the ground plane, which
follows Shi _et al_. [22]. The bins of histogram in each channel are 90. The
model is trained in an end-to-end manner with a batch size of 16. The
optimizer we used is Adam with a learning rate of 0.00005, and $\beta_{1}=0$,
$\beta_{2}=0.999$. Using a 32GB Tesla V100, the training time is about 30
hours for 30 epochs. As for the architectures of DensityNet and RenderNet,
they share most similarities with the networks used in Pix2Pix[6]. More
details about the model architecture and training details can be found in the
supplemental material.

###  4.2 Evaluation Metrics

We use several evaluation metrics to quantitatively assess our results. The
low-level similarity measures include root-mean-square error (RMSE), structure
similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and
sharpness difference (SD), which evaluate the pixel-wise similarity between
two images. We also use high-level perceptual similarity [36] for evaluation
as in previous works. Perceptual similarity evaluates the feature similarity
of generated and real images. We employ the pretrained AlexNet [10] and
Squeeze [5] networks as backbones for evaluation, denoted as $P_{\text{alex}}$
and $P_{\text{squeeze}}$, respectively.

###  4.3 Dataset for Ground View Synthesis

We choose CVUSA [34] and CVACT(Aligned) [22] datasets for comparison in the
central ground-view synthesis setting, following Shi _et al_. [22].
CVACT(Aligned) is a well-posed dataset aligned in Shi _et al_. [22], with 360°
horizontal and 180° vertical visualization in panorama ground truth. Hence, we
selected it for controllable illumination visualization and controllable video
generation. For the dataset CVUSA, we only use it for center-ground view
synthesis as their upper and lower parts of the panoramic image are trimmed
for the geo-localization task, and the number of trimmed pixels is unknown
[34]. During training and testing, we considered the street-view panoramas in
the CVUSA dataset as having a 90° vertical field of view (FoV), with the
central horizontal line representing the horizon. CVACT(Aligned) contains
26,519 training pairs and 6,288 testing pairs, while CVUSA contains 35,532
training pairs and 8,884 testing pairs. We did not choose other available
datasets built in the city scene, such as OP[21] and VIGOR[37], since their
GPS information is less accurate in urban areas compared to open rural areas,
and poorly posed satellite-ground image pairs are not suitable for our task.

(a) input & track (b) Baseline (c) Baseline+Opa (d) Baseline+Illu (e)
Baseline+Illu+Opa (f) Sat2Density

(g) Baseline

(h) Baseline+Opa

(i) Baseline+Illu

(j) Baseline+Illu+Opa

(k) Sat2Density

Figure 3: Ablation study on CVACT (Aligned) dataset. In the first row, the
picture on the upper left is the input image. Each point from left to right is
related to the bottom four rows from up to down. The remaining five images in
the first row are the density rendered from the input satellite image
following the setting (b-f) one by one. The images in the second row are the
satellite depth calculated following the setting (b-f) one by one. ‘Baseline’
means baseline, ‘Opa’ means add non-sky opacity supervision, ‘Illu’ means add
illumination injection, and ‘Sat2Density’ is our final result, compared to
‘Baseline+Illu+Opa’, we concatenate the depth map and initial panorama
together to send to the RenderNet rather than only the initial panorama. The
video could be seen in the project page.

(a) $I_{\text{sat}}$

(b) Pix2Pix [6]

(c) XFork [19]

(d) Shi _et al_. [22]

(e) Sat2Density

(f) Ground Truth

Figure 4: Example images generated by different methods in center panorama
synthesis task. The top two rows show the results on CVACT (Aligned) dataset,
and the bottom two rows show the results on the CVUSA dataset.

###  4.4 Ablation Study

In this section, we conduct experiments to validate the importance of each
component in our framework, including non-sky opacity supervision,
illumination injection, and whether to concatenate depth with the initial
panorama before sending it to the RenderNet. We first present quantitative
comparisons in Table 1 for the center ground-view synthesis setting. It is
evident that the illumination injection most affects the quantitative result,
at the same time, only adding non-sky opacity supervision will lead to a
little drop in the quantitative score. But combining the two approaches will
lead to better scores. Moreover, the comparison on whether concatenate depth
to the RenderNet shows almost equal results in terms of quantitative
comparison.

Figure 3 shows some samples from the rendered panorama video and their
corresponding depth maps. The results show that without the proposed
components (baseline), the rendered depth seems meaningless in the upper half,
while the lower regions look good. We attribute this phenomenon to the fact
that the lower bound of the panorama is the ray that looks down, which is
highly related to the ground region near the shooting center in the satellite.
It can be easily learned by a simple CNN with a simple geometry projection,
which also explains why the work in [22] can render the ground region well.

Compared to the baseline, adding the illumination injection can make the
rendered depth look better, but the trees’ density looks indistinct, and the
sky region’s density is still unclear. While only adding non-sky opacity
supervision, the air region’s opacity turns to zero, but the area between air
and the ground is still barely satisfactory. The supervision did clear the sky
region in the volume density, but the inner region between the sky and the
ground is also smoothed. This is because such coarse supervision cannot help
the model recognize the complex region.

By combining both strategies (Baseline+Illu+Opa), we can achieve a plausible
3D geometry representation that can generate a depth map faithful to the
satellite image and the reference illumination. The volume density is clear
compared to the above settings, and we can easily distinguish the inner
regions.

Furthermore, when depth is incorporated into the rendering process, the
resulting images tend to emphasize regions with depth information. This
reduces the likelihood of generating objects in infinite areas randomly and
leads to a synthesized ground view that more closely resembles the satellite
scene, which can be observed from the video.

Comparison | RMSE $\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SD $\uparrow$ | $P_{alex}\downarrow$ | $P_{squ}\downarrow$
---|---|---|---|---|---|---
Base | 48.40 | 0.4491 | 14.67 | 12.76 | 0.3772 | 0.2486
Base+Opa | 48.39 | 0.4431 | 14.63 | 12.70 | 0.3847 | 0.2525
Base+Illu | 41.62 | 0.4689 | 15.96 | 12.90 | 0.3497 | 0.2225
Base+Opa+Illu | 40.71 | 0.4710 | 16.16 | 12.83 | 0.3329 | 0.2154
Sat2Density | 39.76 | 0.4818 | 16.38 | 12.90 | 0.3339 | 0.2145

Table 1: Ablation study results on CVACT (Aligned) dataset. ‘Base’ means
baseline, ‘Opa’ means add non-sky opacity supervision, ‘Illu’ means add
illumination injection, and ‘Sat2Density’ is our result, compared to
‘Baseline+Opa+Illu’, we concatenate the depth map and initial panorama
together to send to the RenderNet rather than only the initial panorama.

###  4.5 Center Ground-View Synthesis Comparison

In the center ground-view synthesis setting, we compare our method with
Pix2Pix [6], XFork [19], and Shi _et al_. [22]. Pix2Pix and XFork are classic
GAN-based models for image-to-image translation, but ignore the 3D geometry
connections between the two views. Shi _et al_. [22] is the first geometry-
guided synthesis model, which represents the 3D geometry in the depth
probability MPI, showing brilliant results in the center ground-view synthesis
setting.

| Method | RMSE $\downarrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SD $\uparrow$ | $P_{\text{alex}}\downarrow$ | $P_{\text{squeeze}}\downarrow$ |  Inference time/ms
---|---|---|---|---|---|---|---|---
CVACT (Ali.) | Pix2Pix [6] | 49.75 | 0.3852 | 14.38 | 12.09 | 0.4654 | 0.3096 | 10.29
XFork [19] | 48.95 | 0.3710 | 14.50 | 12.32 | 0.4638 | 0.3262 | 17.24
Shi _et al_. [22] | 48.50 | 0.4272 | 14.59 | 12.31 | 0.4059 | 0.2708 | 109.88
Sat2Density | 47.13 | 0.4586 | 14.92 | 12.77 | 0.3842 | 0.2573 | -
Sat2Density-oracle | 39.76 | 0.4818 | 16.38 | 12.90 | 0.3339 | 0.2145 | 33.12
CVUSA | Pix2Pix | 55.27 | 0.2946 | 13.48 | 11.97 | 0.5092 | 0.3902 | -
XFork | 54.11 | 0.2873 | 13.68 | 12.15 | 0.5144 | 0.4041 | -
Shi _et al_. | 53.75 | 0.3451 | 13.75 | 12.06 | 0.4639 | 0.3506 | -
Sat2Density | 53.30 | 0.3301 | 13.78 | 12.38 | 0.4504 | 0.3365 | -
Sat2Density-oracle | 48.75 | 0.3584 | 14.66 | 12.53 | 0.4163 | 0.3058 | -

Table 2: Quantitative comparison with existing algorithms on the CVACT
(Aligned) and CVUSA datasets in center ground-view synthesis setting.
‘Sat2Density’ means we randomly choose nine histograms from the test set when
inference and then average it. ‘Sat2Density-oracle’ means the histogram is
from the GT ground image. The inference time was tested on a Tesla V100.

Figure 5: Controllable illumination synthesis: each row shows the results
rendered from the same satellite image, and each column shares the same
illumination from the same ground image.

#### Quantitative Comparison.

As presented in Table 2, it is evident that Sat2Density achieves the best
performance on all scores, including both low-level and perceptual similarity
measures. Even choosing illumination hint randomly, our model still
outperforms other methods.

Moreover, a combined analysis of the quantitative results of Sat2Density-sin
and controllable illumination in Figure 5 reveals that illumination can
significantly affect both common low-level and perceptual similarity measures,
although the objects in the scene remain unchanged. As a result, it is more
important to consider qualitative comparisons and video synthesis results.

#### Qualitative Comparison.

In Figure 4, we find that the condition-GAN based methods can only synthesize
good-looking ground images, but can not restore the geometry information from
the satellite scene. Shi  _et al_. [22] learn a coarse geometry
representation, so the 3D information in the ground region is more reliable.
For our method, as discussed in the ablation study, the high-fidelity
synthesis (especially in the most challenging regions between the sky and the
ground) is approached by learning faithful density representation of the 3D
space.

###  4.6 Controllable Illumination

As shown in Figure 5, the sky histogram could easily control the image’s
illumination, while the semantics did not change, _e.g_. The road’s color was
changed by giving different illumination, but the shape remains unchanged.
Besides, the illumination interpolation results can be seen on the project
page, which also show the superiority of illumination injection.

###  4.7 Ground Video Generation

In Figure 6, we compare the rendered satellite depth, and synthesized ground
images from a camera trajectory with the expansion of Shi _et al_. [22]. Shi
_et al_. [22] focus on synthesizing ground panorama in the center of the
satellite image, as they learn geometry by depth probability map, we expand
their work by moving the camera position when inference. We also show the
rendered depth maps. It is worth noting that Shi _et al_. [22] cannot generate
a depth map for novel views, due to the intrinsic flaw of the depth
probability representation.

(a) input image & track

(b) Ours sat depth

(c) Shi _et al_. sat depth

(d) Our ground depth

(e) Our ground

(f) Shi _et al_.

Figure 6: Synthesized video & depth comparison. (a) is the input satellite
image, the red curve is the camera trajectory to synthesize video, the red
point is chosen for visualization in (d-f), (b-c) is the rendered satellite
depth by our method and Shi _et al_., (d) is the rendered depth of the ground
image by our method, (e), (f) are the rendered ground image by our method and
Shi _et al_. separately. The video can be seen in the project page.

From the synthesized satellite depth, we observe that Shi _et al_. [22] can
only render a very coarse satellite depth, and is hard to recognize most
regions. In contrast, trees and ground regions can easily distinguish from our
satellite depth, and the depth in ground regions appears smooth. Additionally,
we can render depth in any view direction by volume rendering, as shown in
Figure 6 (d).

Furthermore, we find that the rendered ground video by the expanded Shi _et
al_. [22] has little consistency due to the unfaithful 3D geometry
representation, as evidenced by the inconsistencies present in the trees and
sky. These results demonstrate that Sat2Density is capable of rendering
temporal and spatially consistent videos.

##  5 Discussion and Limitations

Although Sat2Density is advancing the state-of-the-art, it still has some
limitations. For instance, the tree density and visibility of houses are not
perfect in our results, which would come down to the following reasons.
Firstly, the one-to-one satellite and ground image pairs are not optimal for
training, as having multiple ground images corresponding to one satellite
image would be better. Additionally, images taken on different days may
introduce transient objects that our approach is unable to handle. Then, the
projected color map sent to the RenderNet may be too coarse in the region
between sky and ground, which could impact the final result. Finally, well-
aligned image pairs are required to learn geometry, so we are unable to
evaluate the effectiveness of our approach in city scenes for more coarse GPS
precision in the city. Therefore, while our work is a promising start for
learning geometry from cross-view image pairs, there are still many challenges
that need to be addressed.

##  6 Conclusion

In this paper, we propose a method, _i.e_. Sat2Density, to learn a faithful 3D
geometry representation of satellite scenes from satellite-ground image pairs
through the satellite-to-ground-view synthesis task. Our approach tackles two
critical issues, the infinity issue and the illumination difference issue, to
make geometry learning possible. By leveraging the learned density, our model
is capable of synthesizing spatial and temporal ground videos from satellite
images even with only one-to-one satellite-ground image pairs for training. To
the best of our knowledge, our method represents the first successful attempt
to learn precise 3D geometry from satellite-ground image pairs, which
significantly advances the recognition of satellite-ground tasks from a
geometric perspective.

#### Acknowledgements.

This work was supported by the National Nature Science Foundation of China
under grant 62101390.

## References

  * [1] Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, et al.  Gaudi: A neural architect for immersive 3d scene generation.  arXiv:2207.13751, 2022.
  * [2] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J. Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3d generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 16102–16112, 2022.
  * [3] Terrance DeVries, Miguel Ángel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind.  Unconstrained scene generation with locally conditioned radiance fields.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 14284–14293, 2021.
  * [4] Zekun Hao, Arun Mallya, Serge J. Belongie, and Ming-Yu Liu.  Gancraft: Unsupervised 3d neural rendering of minecraft worlds.  In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 14052–14062, 2021.
  * [5] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and $<$ 0.5mb model size.  arXiv:1602.07360, 2016.
  * [6] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 5967–5976, 2017.
  * [7] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 9906, pages 694–711, 2016.
  * [8] James T. Kajiya and Brian Von Herzen.  Ray tracing volume densities.  In ACM SIGGRAPH computer graphics, pages 165–174, 1984.
  * [9] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  In Advances in Neural Information Processing Systems, pages 852–863, 2021.
  * [10] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, pages 1106–1114, 2012.
  * [11] Zuoyue Li, Zhenqiang Li, Zhaopeng Cui, Rongjun Qin, Marc Pollefeys, and Martin R. Oswald.  Sat2vid: Street-view panoramic video synthesis from a single satellite image.  In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 12416–12425, 2021.
  * [12] Xiaohu Lu, Zuoyue Li, Zhaopeng Cui, Martin R. Oswald, Marc Pollefeys, and Rongjun Qin.  Geometry-aware satellite-to-ground image synthesis for urban areas.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 856–864, 2020.
  * [13] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 7210–7219, 2021.
  * [14] Nelson L. Max.  Optical models for direct volume rendering.  IEEE Trans. Vis. Comput. Graph., 1(2):99–108, 1995.
  * [15] Radu Paul Mihail, Scott Workman, Zach Bessinger, and Nathan Jacobs.  Sky segmentation in the wild: An empirical study.  In IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–6, 2016.
  * [16] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 12346, pages 405–421, 2020.
  * [17] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2337–2346, 2019.
  * [18] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2337–2346, 2019.
  * [19] Krishna Regmi and Ali Borji.  Cross-view image synthesis using conditional gans.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 3501–3510, 2018.
  * [20] Krishna Regmi and Ali Borji.  Cross-view image synthesis using geometry-guided conditional gans.  Comput. Vis. Image Underst., 187, 2019.
  * [21] Krishna Regmi and Mubarak Shah.  Bridging the domain gap for ground-to-aerial image matching.  In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 470–479, 2019.
  * [22] Yujiao Shi, Dylan Campbell, Xin Yu, and Hongdong Li.  Geometry-guided street-view panorama synthesis from satellite imagery.  IEEE Trans. Pattern Anal. Mach. Intell., 44(12):10009–10022, 2022\.
  * [23] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li.  Spatial-aware feature aggregation for image based cross-view geo-localization.  Advances in Neural Information Processing Systems, 32, 2019.
  * [24] Yujiao Shi, Liu Liu, Xin Yu, and Hongdong Li.  Spatial-aware feature aggregation for image based cross-view geo-localization.  In Advances in Neural Information Processing Systems, pages 10090–10100, 2019.
  * [25] Yujiao Shi, Xin Yu, Liu Liu, Tong Zhang, and Hongdong Li.  Optimal feature transport for cross-view image geo-localization.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 11990–11997, 2020.
  * [26] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J. Corso, and Yan Yan.  Multi-channel attention selection GAN with cascaded semantic guidance for cross-view image translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 2417–2426, 2019.
  * [27] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 8798–8807, 2018.
  * [28] Felix Wimbauer, Nan Yang, Christian Rupprecht, and Daniel Cremers.  Behind the scenes: Density fields for single view reconstruction.  CoRR, abs/2301.07668, 2023.
  * [29] Scott Workman and Hunter Blanton.  Augmenting depth estimation with geospatial context.  In Proceedings of IEEE International Conference on Computer Vision (ICCV), pages 4542–4551, 2021.
  * [30] Scott Workman, Muhammad Usman Rafique, Hunter Blanton, and Nathan Jacobs.  Revisiting near/remote sensing with geospatial attention.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 1768–1777, 2022.
  * [31] Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang.  Sinnerf: Training neural radiance fields on complex scenes from a single image.  In Proceedings of the European Conference on Computer Vision (ECCV), volume 13682, pages 736–753, 2022.
  * [32] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4578–4587, 2021.
  * [33] Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs.  Predicting ground-level scene layout from aerial imagery.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4132–4140, 2017.
  * [34] Menghua Zhai, Zachary Bessinger, Scott Workman, and Nathan Jacobs.  Predicting ground-level scene layout from aerial imagery.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 4132–4140, 2017.
  * [35] Jiaming Zhang, Kailun Yang, Chaoxiang Ma, Simon Reiß, Kunyu Peng, and Rainer Stiefelhagen.  Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 16896–16906, 2022.
  * [36] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 586–595, 2018.
  * [37] Sijie Zhu, Taojiannan Yang, and Chen Chen.  VIGOR: cross-view image geo-localization beyond one-to-one retrieval.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), pages 3640–3649, 2021.

## Appendix

In the appendix, we first show our motivation and then present the details of
the model architecture and training process of our Sat2Density model. After
that, we describe the satellite and ground-view panorama camera models. Last,
we give more discussion about our work for a better understanding of our paper
and hope to advance viewing remote sensing and ground imagery from a geometric
perspective. Code, pre-trained models, and more video results can be found on
our project page.

##  A Motivation

Sat2Density focuses on the geometric nature of generating high-quality ground
street views conditioned on satellite images learning from collections of
satellite-ground image pairs. The long-suffered issue from the unknown 3D
information is addressed by separating the sky/non-sky regions with reasonable
3D density volumes learned. We believe our new perspective on the longstanding
yet challenging problem of satellite-ground novel view synthesis would bring
more insights for a wide range of 3D vision tasks, including but not limited
to (1) using satellite images for autonomous driving with faithful 3D
geometry, (2) providing promising and novel solutions for visual localization
with satellite images.

##  B Addition Implementation Details

###  B.1 DensityNet

The DensityNet is taken from the generator of Pix2Pix [6]. Compared to vanilla
Pix2Pix in PyTorch implementations from pix2pix in PyTorch, our generator
replaces the activation function in the initial layer and downsample layers
from ReLU to PReLU, sets the number of res-block to 6, and replaces ReLU with
Tanh in the last layer. The final output of DensityNet is an explicit volume
density $V_{\sigma}\in\mathbb{R}^{H\times W\times N}$, rather than predicting
an image with resolution $H\times W\times 3$.

###  B.2 Illumination Injection

To inject the illumination, we first calculate the RGB histogram of the sky
region in the ground image with 90 bins in each color channel. Following the
way process style in GANcraft[4], we use a style encoder to predict a style
code, then use an MLP that is shared across all the style conditioning layers
to convert the input style code to an intermediate illumination feature. The
key difference is that the input of the style encoder is a histogram rather
than an image. When inference, we could randomly select a histogram as the
illumination input, at the same time, interpolation in the $z$ space between
two histograms is also allowed. The interpolation visualization video result
can be seen on the project page.

###  B.3 RenderNet

The RenderNet is a variation of Pix2Pix [6]. As shown in Figure 7, the key
difference is that we inject the style feature in the last three Upsample
blocks, which includes the illumination information of the groundtruth image
during training, thereby mitigating the effects of illumination changes.

Figure 7: The architecture of RenderNet. We inject the illumination feature in
the decoder.

###  B.4 Discriminator

The discriminator we use is a multi-scale discriminator that differs from the
vanilla multi-scale discriminator used in pix2pixHD [27]. While the vanilla
discriminator operates on images of different scales, we use three
discriminators: $D_{1}$, $D_{2}$, and $D_{3}$. $D_{1}$ works on panorama
images, while $D_{2}$ and $D_{3}$ operate on perspective images obtained by
randomly sampling from the input panorama using a perspective transformation,
but at different scales. The two discriminators operate on perspective images
because the distortion on the upper and lower bounds of the panorama is
challenging for the convolution layer. Specifically, the field of view (FOV)
of the sampled perspective images is 100. In our ablation study, all results
use the same multi-scale discriminator. The input image size for $D_{1}$,
$D_{2}$, and $D_{3}$ is $64\times 256$, $64\times 64$, and $32\times 32$
respectively.

###  B.5 Additional Training Details

The weight for L1 loss, L2 loss, KL loss, feature matching loss, perceptual
loss, $\mathcal{L}_{\rm snop}$ and GAN loss are 1, 10, 0.1, 10, 10, 1, 1
respectively when training. In volume rendering, we sample 100 points along
each ray.

##  C Satellite and Panorama camera model

Actually, there are no given camera instincts in the original CVUSA and CVACT
datasets, which only contain image pairs collected from Google Earth in the
same location by GPS, we follow the assumptions in Shi _et al_. [22], which
assumes that satellite images show the top of objects in an overhead view,
which approximates parallel projection, while street-view panoramas capture
scenes at ground level with a spherical equirectangular projection.

To describe a panoramic image with a 360-degree horizontal and 180-degree
vertical field of view, we use the equirectangular projection and spherical
coordinate system. To accomplish this, we assign the camera location as
$\mathbf{o}$, and the width and height of the panorama image as $w$ and $h$,
respectively. We use $x$ and $y$ as the pixel coordinates of the image pixel
under consideration, and then we can use the following equations to determine
the azimuthal and zenith angle $\theta$ and $\phi$, respectively:

$\theta=\frac{2\pi x}{w},\phi=\frac{\pi y}{h}$

The equation allow us to determine the view direction $\mathbf{d}$ through any
given image pixel.

We illustrate the orientation corresponding to the CVACT (align) dataset in
Figure 8, where the same color indicates the same direction.

(a) satellite (b) panorama

Figure 8: Here is an example of an aligned satellite and ground panorama image
pair from the training dataset. In the satellite image, the north direction is
upward, while in the ground panorama image, the central column line represents
the north direction. Both display the same red color. The central horizontal
line in the panorama corresponds to the horizon.

##  D Discussion

###  D.1 Urban scenes

The nadir satellite image can not see the vertical surfaces of tall buildings.
Except for the issue of unseen vertical surfaces, two representative cases of
tall buildings and transient objects (_e.g_., cars) will challenge our method,
though we believe the geometric perspective would facilitate the task for
urban scenes.

###  D.2 Infinite region

We assume that objects beyond the top view coverage in street view images only
include the sky. It is hard to find out which object (e.g. tree) lies outside
of the satellite scene, for we have no real 3D shape to find it. Nevertheless,
our assumption has shown clear effects, as evident from the ablation study
(check in the project page video).

###  D.3 Assumption on horizontal ground planes

Sat2Density has the horizontal assumption as we did not know the camera
location and world coordinate system. Another assumption we used is that the
“world” is finite and limited by the satellite image. The used datasets follow
these two assumptions and provide 1-to-1 paired data. Given by these facts,
the movement of cameras is indeed on the ground plane with a constant height
(_e.g_., 2m in our method and prior arts like [22]). From the perspective of
view synthesis, such assumptions should work well, but it will inherently lead
to inaccurate 3D scene geometry when the whole ground region of a scene is
sloped. To resolve this problem, further studies could be explored with some
new problems: (1) how to estimate the slope from a single orthogonally-
rectified satellite image, and (2) how to define a proper world coordinate
system and place the ground-view camera(s) in the world.