# Multichannel Social Signatures and Persistent Features of Ego Networks

S. Heydari  Department of Computer Science, Aalto University, P.O. Box 14400,
FI-00076 AALTO, Finland<EMAIL_ADDRESS>S.G.B. Roberts  School of
Natural Sciences and Psychology, Liverpool John Moores University, Byrom
Street, Liverpool L3 3AF, UK  R.I.M. Dunbar  Department of Experimental
Psychology, University of Oxford, Oxford OX1 3UD, UK  J. Saramäki  Department
of Computer Science, Aalto University, P.O. Box 14400, FI-00076 AALTO, FInland
Helsinki Institute of Information Technology HIIT, Aalto University, Finland

###### Abstract

The structure of egocentric networks reflects the way people balance their
need for strong, emotionally intense relationships and a diversity of weaker
ties. Egocentric network structure can be quantified with ’social signatures’,
which describe how people distribute their communication effort across the
members (alters) of their personal networks. Social signatures based on call
data have indicated that people mostly communicate with a few close alters;
they also have persistent, distinct signatures. To examine if these results
hold for other channels of communication, here we compare social signatures
built from call and text message data, and develop a way of constructing mixed
social signatures using both channels. We observe that all types of signatures
display persistent individual differences that remain stable despite the
turnover in individual alters. We also show that call, text, and mixed
signatures resemble one another both at the population level and at the level
of individuals. The consistency of social signatures across individuals for
different channels of communication is surprising because the choice of
channel appears to be alter-specific with no clear overall pattern, and ego
networks constructed from calls and texts overlap only partially in terms of
alters. These results demonstrate individuals vary in how they allocate their
communication effort across their personal networks and this variation is
persistent over time and across different channels of communication.

##  I Introduction

Social relationships that are strong and supportive are fundamentally
important for health and well-being, in both humans and other primates
house1988 ; lyubomirsky2005 ; wittig2008 ; holtlunstad2010 ; manninen2017 .
While close, emotionally intense relationships provide support and cohesion,
weaker ties have been associated with the benefits of diversity and access to
resources outside one’s everyday social circles Granovetter ; Burt . At the
same time, maintaining social ties comes at a cost: time and cognitive
resources are finite miritello2013time ; miritello2013limited . This cost is
particularly high for close relationships Roberts2009 . As a result, personal
networks typically have only a few close ties and many weak ties. This is
visible both at the level of entire social networks Onnela2007  as well as in
how people structure their personal networkssaramaki2014persistence .

The way people balance their need for strong, cohesive ties and weak ties that
lead outside their closest network is reflected in so-called _social
signatures_. The social signature of an ego measures the fraction of
communication targeted at alters of each rank, when the alters are ranked
according to this fraction. Social signatures therefore quantify rank-
frequency relationships of alters in egocentric networks. In Ref.
saramaki2014persistence , it was shown that people place their mobile
telephone calls to their alters very unevenly across their ego networks, so
that a few closest alters get a disproportionate fraction of calls. This is
reflected in social signatures that typically decay slower than exponentially.
It was also shown in Ref. saramaki2014persistence  that each individual has
their own, distinctive social signature that persists in time, even when there
is a large amount of turnover in the ego network. Similar observations were
made in Ref. centegheller2017  with a different dataset on mobile telephone
calls.

However, social relationships are shaped and maintained through a diversity of
communication channels Vlahovic2012 ; reid2005 ; nanavati2008 ; wang2013 ;
zignani2014 ; quadri2014 . People do not use these channels uniformly –
rather, the choice of channel depends on many factors. These include the type
of relationship (nature of social tie), general channel preferences, the time
of the event (social norms) and the reason for communicating; see, e.g.,
reid2005  on why texters text. To examine if the properties of social
signatures are generalizable and genuine features of egocentric networks, it
is therefore important to look at data from multiple channels of
communication, both separately and together. Combining information on
different channels can, however, be problematic because of their intrinsic
differences. For example, the number of calls or their total duration is
typically used as a proxy for tie strength in mobile telephone call data
saramaki2015seconds . But text messages, another common form of communication
via mobile devices, have no duration, and the number of text messages between
an ego-alter pair is not directly comparable to the number of calls between
that pair. While one call can be thought to represent one conversation, one
text message is typically only a part of a longer conversation.

In this paper, we study social signatures that are based on calls, texts, and
both. To this end, we develop a way of constructing weighted ego networks from
time-stamped communication data that makes different channels more comparable
(see Fig. 1), and also allows for the construction of mixed social signatures
based on both call and text message data. We apply this method to two data
sets on mobile telephone communication, and observe that both single-channel
and mixed signatures are persistent over time, as observed earlier for calls-
only signatures. We also observe that the call and text signatures are
surprisingly similar for each ego. This is unexpected, because at the same
time, the call and text networks of most egos overlap only partially, and
there are no clear patterns of channel preference: the choice of channel
appears independent of alter rank in mixed signatures.

##  II Materials and Methods

###  II.1 Datasets

Table 1: The two data sets used in this study. NCPM = number of calls per user per month; NTPM = number of text messages per user per month. | DS 1 | DS 2
---|---|---
Number of active users | 506330 | 24
Length of data-collection period | 7 months | 18 months
10-percentile of NCPM | 43 | 67
Median of NCPM | 83 | 127
90-percentile of NCPM | 194 | 278
10-percentile of NTPM | 20 | 105
Median of NTPM | 45 | 317
90-percentile of NTPM | 131 | 2019

We use two data sets of mobile telephone calls and text messages (see Table
1). _Data set DS1_ comprises the Call Detail Records (CDRs) for calls and text
messages of the anonymized customers of a mobile operator in an European
country, collected over 7 months (see, e.g., Karsai2011 ; kivela2012multiscale
). We applied an activity threshold and retained only users with more than
$20$ calls and more than $7$ text messages per month, retaining 506,330 users.
_Data set DS2_ contains the times and recipients of outgoing calls and text
messages for 24 students in the UK Roberts11 ; saramaki2014persistence . The
data collection period is 18 months, during which the students graduated from
high school and moved on to University or work.

As our aim was to construct social signatures and study their persistence in
time, we divided both data sets into two equal-sized consecutive time
intervals (3.5 months each for DS1 and 9 months for DS2); this was for being
able to compare the stability of the shapes of the signatures for the first
and second halves. The choice to split into two was merely for convenience;
please note that in saramaki2014persistence , DS2 was analyzed using three
intervals, yielding similar results for calls.

###  II.2 Constructing egocentric networks and social signatures

Social signatures are calculated from weighted egocentric networks, where the
link weights represent the amount of communication between the focal ego and
the ego’s alters. Social signatures measure the fraction of communication to
alters of each rank, when the alters are ranked according to this fraction. In
Ref. saramaki2014persistence , the number of outgoing calls that took place
during the data collection period were used as weights. However, when there
are multiple channels, the question of how to define weights is not
straightforward. The simplest solution would be to use the number of
communication events as the weight for all channels. However, this is
problematic. In our case of calls and texts, as disussed above the numbers of
calls and texts cannot be directy compared. One call can be associated with
one conversation, while one conversation by texting may amount to a large
number of individual text messages.

Here, our aim is to make the channels more comparable by focusing on their
timelines and coarse-graining events in time. We do this as follows: we take
the timeline of each ego-alter link, and divide it into time bins of one hour.
Note that one hour has been chosen for convenience and to be clearly longer
than the time scale of tens of seconds to minutes associated with correlated
calls or texts Karsai2011 ; backlund2014 ). Then, for both calls and texts, we
count the number of bins which contain at least one communication event (see
Fig. 1A). Thus we count the number of one-hour time bins in which at least one
communication activity takes place. These counts are then used as link weights
for the egocentric networks: _e.g._ , a weight of $w=5$ indicates that there
were five hours where there was call activity with the alter. Calls that begin
in one time bin and stretch along several time bins contribute accordingly to
several units of weight. Defining link weights on the basis of time bins also
makes it possible to construct _mixed_ link weights, as one can count the
number of time slots where either at least one call OR one text message took
place. An advantage of this method is that it can be used to calculate link
weights that quantify the amount of communication or social interaction in any
channel, as long as the time stamps of interaction events are available for
each ego-alter link.

With the time-bin-based weights, social signatures are calculated as in
saramaki2014persistence : for each egocentric network, alters are ranked
according to their link weight, and the fraction of link weight out of the sum
of all link weights is computed as function of alter rank. The social
signature of ego $i$ then reads

$\sigma_{i}=\\{(w_{i1}/\sum_{j=1}^{k_{i}}{w_{ij}}),\ldots,(w_{ik_{i}}/\sum_{j=1}^{k_{i}}{w_{ij}})\\},$
(1)

where the alters $j$ are sorted by weight in decreasing order and $k_{i}$ is
the degree (number of alters) of $i$.

Figure 1: Constructing egocentric networks from calls and texts using time-
binned weights. A) The timelines corresponding to each of the ego’s alters are
divided into bins–we use bins that span one hour. Then, the number of bins
with at least one communication event is computed. These numbers are used as
link weights for egocentric networks (panel B). For the mixed networks, the
link weights represent the number of bins where either calls, texts, or both
are taking place.

###  II.3 Comparing Social Signatures

In order to determine the persistence of social signatures in time, a way of
comparing their shapes and measuring the similarity or difference between two
given signatures is needed. In (saramaki2014persistence ), the Jensen-Shannon
divergence lin1991divergence  (JSD) was used for comparing pairs of social
signatures and we also use the JSD in this analyses. The JSD is defined as:

$JSD_{(\sigma_{1},\sigma_{2})}=H(\frac{1}{2}\sigma_{1}+\frac{1}{2}\sigma_{2})-\frac{1}{2}[H(\sigma_{1})+H(\sigma_{2})],$
(2)

where $\sigma_{1}$ and $\sigma_{2}$ are two social signatures, as defined in
Eq. 1, and $H(\sigma)$ is the Shannon entropy of $\sigma$.

The Jensen-Shannon divergence is a generalized form of the Kullback-Leibler
divergence. The square root of the Jensen-Shannon divergence can be used as a
distance function. Because the JSD can deal with zero probabilities, it allows
us to compare social signatures of different lengths, that is, signatures
computed from egocentric networks with different numbers of alters, $k_{1}\neq
k_{2}$. To compare two signatures of lengths $k_{1}$ and $k_{2}$ where
$k_{2}>k_{1}$, we append zero entries ($w_{ij}=0$) to the shorter signature
for $k_{1}<j\leq k_{2}$ so that both signatures are of equal length.

The overlap of two sets of alters in a pair of egocentric networks can be
measured by the Jaccard coefficient

$J(\sigma_{1},\sigma_{2})=\frac{|\sigma_{1}\cap\sigma_{2}|}{|\sigma_{1}\cup\sigma_{2}|},$
(3)

where $\sigma_{1}$ and $\sigma_{2}$ are the social signatures corresponding to
the networks. As an example, the Jaccard coefficient between the call
signature and text signature of an ego is defined as the number of alters the
ego has contacted by both call and text in the period divided by number of
alters the ego has contacted by call or text. If there is complete overlap
between the alters contacted by call and text, then J = 1. If there is no
overlap between alters contacted by call and text, then J = 0.

##  III Results

###  III.1 Single-Channel and Mixed Signatures Are Persistent

We begin our analysis by demonstrating that all three types of signatures –
call, text, and mixed – are persistent at the level of individuals, as was
shown for call-based signatures for DS2 in Ref. saramaki2014persistence .
Here, we define persistence as the social signature retaining its shape over
time, with individual level variation in JSD that is smaller than the average
JSD between signatures in the whole population.

To examine this persistence, we divide the data collection periods of the two
sets into two intervals of 3.5 and 9 months for sets DS1 and DS2,
respectively. We then calculate the weighted egocentric networks for each ego
in each interval. As explained in detail above, we use the number of one-hour
time bins with calls, texts, or either for determining the link weights
between the ego and alters. We compute the social signatures for each
egocentric network and each interval by ranking alters according to their
weight and calculating the fraction of weight at each rank. Following Ref.
saramaki2014persistence , we then calculate self-distances by computing the
JSD between an ego’s own signatures in consecutive intervals. We also
calculate reference distances by computing the average JSD between the
signature of the ego and those of all other egos. We repeat these calculations
for both channels (calls and texts) as well as mixed networks (calls and
texts).

The distributions of self and reference distances of call, text, and mixed
signatures are displayed in Fig. 2. For all three types of signatures and for
both data sets, the bulk of the distribution of self-distances clearly lies
below the reference distances. Self-distances are on average smaller than the
distance between the signatures of two random egos, and there is less spread
in their distribution. These differences in the distributions of Fig. 2
indicate that the changes of an ego’s signature in time are smaller than the
variation of signature distances in the population, whether calculated from
calls or texts or both. This means that the individual differences in
signature shapes are a real feature of the egocentric networks instead of
random variation resulting in noisy, unstable signatures. The persistence of
social signatures is therefore not only a feature of egocentric networks built
from phone call data, but a more general phenomenon.

###  III.2 Single-Channel and Mixed Signatures Have Similar Shapes, Even at
the Ego Level

We have now established that the three types of signatures are persistent
characteristics of egocentric networks. Next, we compare the shapes of these
signatures, first at the population level and then at the level of
individuals. It was shown in Ref. saramaki2014persistence  that call
signatures in DS2 are rather skewed: a small number of top-ranking alters get
a disproportionate share of communications. We find that all three types of
signatures show this skewed shape at the level of individuals and at the
population level. This is seen in Fig. 3 that shows the three types of
signatures of one person (a) and the population-averaged signatures (b).

It also appears that the two types of single-channel signatures are more
similar for each ego than they are between egos–the shapes of the call and
text signatures of an ego look similar. This is confirmed by use of the JSD.
We calculated the self-distances between an ego’s call and text signature as
well as reference distances between all pairs of call and text signatures,
aggregated over the entire period of observation. The resulting distributions
for both data sets again indicate that self-distances are on average smaller
than reference distances (Fig. 3). Even though the difference is slightly less
pronounced than for distances of the same signature type between different
intervals (Fig. 2), the shapes of call and text signatures of an ego appear to
correlate.

###  III.3 Single-Channel Egocentric Networks Differ in Composition

The similarity in the shapes of the call and text signatures of each ego would
be expected if their call and text networks were similar and included the same
alters with similarly ranked weights. However, this is not the case: the call
and text networks of an ego are typically different. Instead of the same
alters appearing in both networks, many alters are only called or texted, and
therefore included in one network only. This is in line with literature on
network-level differences nanavati2008 ; wang2013 ; zignani2014 .

This can be seen for both datasets in the distributions of Jaccard indices in
Fig. 4 a) and b), computed for the sets of called and texted alters of each
ego. The values of the Jaccard indices are mostly low. This means that while
some alters are in both networks, most alters are not. Also, as seen in the
lower panels (c,d) of Fig. 4, the ranks of those alters who are present in
both call and text networks correlate only moderately: an alter who is among
the most called alters may receive a far smaller share of text messages.

###  III.4 Channel Choice Does not Depend on Alter Rank

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2:  Social signatures are persistent at the individual level. This
holds for both channels (calls and texts) as well as mixed signatures
combining both. Panels (a), (c), (e): Dataset 1 (the large dataset), Panels
(b), (d), (f): Dataset 2 (students). The distributions of distances between
social signatures of each ego in two consecutive equal-sized intervals are
shown in blue (self–distances). The reference distributions of distances
between signatures of different egos are shown in red (reference distances).
Comparing the distributions of self–distances with reference distances
verifies the persistence of call, text and mixed signatures, as self-distances
are on average smaller than reference distances.

Next, we investigated whether there are systematic differences in the call and
text networks of egos; such differences might explain the signature shapes and
their similarities, despite call and text networks being different. To this
end, we take a look at mixed egocentric networks calculated using both calls
and texts, and investigate their weight composition. We focus on the share of
calls and texts for each rank; note that since we are counting time slots,
there are slots with both channels present.

One example mixed signature and its weight composition are shown in Fig. 5. It
appears that there is no clear pattern, except perhaps an slightly increased
focus on calls around ranks 11-16. A likely explanation is that the choice of
communication channel depends on features of the relationship in question
other than its emotional closeness that correlates with ranks. This is
supported by Fig. 6 that shows the shares of text messages in all ego-alter
relationships of DS2 (top) and a large sample of ego-alter relationships of
DS1 (bottom). In DS2, the only systematic feature is that alters at top ranks
typically receive both calls and text messages, and the fraction of text-only
and call-only relationships increases towards the lower ranks. Top
relationships appear more balanced regarding communication channels in DS1
too. Beyond that, there are no systematic changes that depend on alter rank.

(a)

(b)

(c)

(d)

Figure 3: The similarities of social signatures of different types. Panel (a)
shows the call, text and mixed signatures of one person in the Dataset 1. The
three signatures look similar. Panel (b) illustrates the average signatures
over the population in Dataset 2. The population-level signatures are also
fairly similar. Panels (c) and (d) compare the distance distributions of the
call and text signatures of same egos with the distributions of call and text
signatures of different people as a reference. The call and text signatures of
each ego are more similar than pairs of signatures of different people.

(a)

(b)

(c)

(d)

Figure 4:  Although the shapes of call and text signatures of an ego are
relatively similar to each other, the egocentric networks formed through
different channels are different in the membership and ranking of alters. The
distribution of Jaccard indices between the sets of call and text alters are
shown in (a) for DS1 and (b) for DS2. The distribution of correlation
coefficients between call ranks and text ranks of those alters who are in both
networks is shown in (c) for DS1 and in (d) for DS2. Alters who are only in
one of the networks are not considered. Figure 5: Top panel: an example mixed
signature for one of the egos in DS2. Bottom panel: fractions of calls and
texts for the same ego, for each rank. The red and blue areas of the bars
represent the fractions of texts and calls, respectively, of the link weight
between the ego and the alter. The purple areas represent the fraction of time
slots with both calls and texts.

(a)

(b)

Figure 6: Top (a): Each dot shows the fraction of text slots in an ego-alter
relationship as a function of rank, for the smaller DS2. The plot contains all
ego-alter pairs. There appears to be no general pattern, except that the top
ranks are mostly occupied by alters who are both called and texted, while in
the tails of the signature (ranks $>$10 or so), there are more alters who are
only texted or called. Bottom (b): A heat-map version of the top panel for
DS1, with colors indicating the number of ego-alter pairs with a given
fraction of texts at each rank in DS1. In this dataset, texts are used much
less than in DS2.

##  IV Discussion

Social signatures quantify how people allocate their communication across the
members of their personal networks. In this paper, we used two separate
datasets to explore how social signatures based on calls, texts or both vary
over time and across individuals. There were three key findings. First,
individuals vary in how they allocate their time across their ego networks and
this variation is persistent over time, despite a turnover of individuals
alters in the network. This finding was initially reported in Ref.
saramaki2014persistence  using a small sample of 24 students of a similar age
going through the specific transition from school to University or work (DS2
in this paper); Ref. centegheller2017  confirmed the finding for a data set of
$N=93$. The current paper replicates this finding in a much larger, more
demographically diverse sample of over 500,000 people. Second, this individual
variation in social signatures was present across different channels of
communication - phone calls, text messages and a combined network based on
both calls and texts. This is despite the fact that there was little overlap
in the individuals who were called and texted. Third, regardless of the
channel, the top alters get a disproportionately large fraction of all
communication. This is seen in the shapes of three types of signatures (calls,
texts, and mixed). Thus individual variation in social signatures does not
appear to depend on the channel of communication or the specific alters in the
network at a particular point in time, but instead reflects a stable
characteristic of how individuals distribute their communication effort across
their personal networks.

Why are the social signatures from different channels similar? One could
envision an underlying complete egocentric network with tie strengths that
measure the closeness of all the relationship an ego maintain with their
alters. Within this network there is a distribution of tie strengths where
there are few strong and many weak ties dunbar2018anatomy . Then observations
on one channel of communication would be incomplete samples of the underlying
complete network (see, e.g., nanavati2008 ; wang2013 ; zignani2014  for
studies on network-level differences between calls and texts). Individuals
differ in how they allocate their communication across their network, some
individuals allocating a greater fraction of communication to a smaller number
of alters and others allocating communication more evenly across their
network. Thus constructing the social signatures based on different channels
of communication would still pick up this individual variation, even if the
specific alters detected by the different channels of communication vary.
There may also be less fundamental reasons: communication habits, memory
effects, or similar. However, it has been shown that for calls, alter ranks do
correlate with emotional closeness saramaki2014persistence , which supports
the first explanation. To fully understand which sample of alters is captured
by different channels of communication, further research is needed on how
people use different channels of communication to maintain their set of
relationships to family and friends, and how this communication is related to
the underlying tie strength of the relationship reid2005 ; Vlahovic2012 ;
wiese2015you .

The finding that individuals have social signatures that are stable over time
and persist despite the turnover of individual alters has now been shown in a
number of samples from different countries and across different channels of
communication including phone calls saramaki2014persistence ; centegheller2017
, text messages and combined call and text networks (this study) and email
godoy2016long . Given the robustness of this finding, further research is now
needed on the causes and consequences of individual variation in social
signatures. Whilst everyone is subject to similar fundamental time and
cognitive constraints on sociality dunbar2018anatomy , the way people choose
to allocate their communication effort across their networks shows stable
individual variation. Some of this individual variation appears to be due to
personality characteristics centegheller2017 , which are also broadly stable
over time caspi2005personality . Other characteristics that may be associated
with individual variation in social signatures are age and gender, which are
linked with variation in communication patterns bhattacharya2016sex  and
friendship styles roberts2015managing . Further, given the importance of
social relationships to health and well-being house1988 ; lyubomirsky2005 ;
wittig2008 ; holtlunstad2010 ; manninen2017  individual variation in social
signatures may have consequences for outcomes such as stress and loneliness.
Whilst all people distribute their communication very unevenly across their
network saramaki2014persistence , some people focus an even greater proportion
of their communication on a smaller number of alters. Further research could
examine how these different patterns of time allocation across the network are
linked to well-being, particularly during times of network change which put
pressure on the time required to maintain relationships, such as the
transition to University Roberts11  or entering into romantic relationships
milardo1983developing . Further, it would be important to see that our results
can be replicated with other data sets containing calls and texts; as usual
for this kind of data, our data cannot be made public because of privacy
reasons. To conclude, this study demonstrated using two separate samples that
there is individual variation in the way people allocate their time across
their social networks, and these social signatures are persistent over time
and across different channels of communication.

##  V Declarations

Availability of data and material. The source data used in this study cannot
be made publicly available because of privacy restrictions. Ref.
saramaki2014persistence  has some Supplementary Data in relation to DS2.

Competing interests. The authors declare that they have no competing
interests.

Funding. SH and JS acknowledge funding from the Academy of Finland, project
n:o 297195.

Acknowledgements. We thank A.-L. Barabási for providing data set DS1.

Author contributions. SGBR and RIMD conceived and performed the experiments;
SH and JS analyzed data; SH, JS, SGBR, and RIMD wrote the paper.

## References

  * (1) J. House, K. Landis, and D. Umberson, “Social relationships and health,” Science, vol. 241, pp. 540–545, 1988.
  * (2) S. Lyubomirsky, L. King, and E. Diener, “The benefits of frequent positive affect: Does happiness lead to success?,” Psychological Bulletin, vol. 131, pp. 803–855, 2005.
  * (3) R. Wittig, C. Crockford, J. Lehmann, P. Whitten, R. Seyfarth, and D. Cheney, “Focused grooming networks and stress alleviation in wild female baboons,” Hormones and Behavior, vol. 54, pp. 170–177.
  * (4) J. Holt-Lunstad, T. Smith, and J. Layton, “Social relationships and mortality risk: A meta-analytic review,” PLoS Medicine, vol. 7, p. e1000316, 2010\.
  * (5) S. Manninen, L. Tuominen, R. Dunbar, T. Karjalainen, J. Hirvonen, E. Arponen, R. Hari, I. Jääskeläinen, M. Sams, and L. Nummenmaa, “Social laughter triggers endogenous opioid release in humans,” The Journal of Neuroscience, vol. 37, pp. 6125–6131, 2017.
  * (6) M. S. Granovetter, “The strength of weak ties,” American Journal of Sociology, vol. 79, pp. 1360–1380, 1973.
  * (7) R. S. Burt, Structural Holes: The Social Structure of Competition.  Harvard University Press, 1995.
  * (8) G. Miritello, E. Moro, R. Lara, R. Martínez-López, J. Belchamber, S. G. Roberts, and R. I. Dunbar, “Time as a limited resource: Communication strategy in mobile phone networks,” Social Networks, vol. 35, no. 1, pp. 89–95, 2013.
  * (9) G. Miritello, R. Lara, M. Cebrian, and E. Moro, “Limited communication capacity unveils strategies for human interaction,” Scientific reports, vol. 3, 2013.
  * (10) S. G. B. Roberts, R. I. M. Dunbar, T. V. Pollet, and T. Kuppens, “Exploring variation in active network size: Constraints and ego characteristics,” Social Networks, vol. 31, no. 2, pp. 138–146, 2009.
  * (11) J. P. Onnela, J. Saramäki, J. Hyvönen, G. Szabó, D. Lazer, K. Kaski, J. Kertész, and A. L. Barabási, “Structure and tie strengths in mobile communication networks,” Proceedings of the National Academy of Sciences, vol. 104, no. 18, pp. 7332–7336, 2007.
  * (12) J. Saramäki, E. A. Leicht, E. López, S. G. Roberts, F. Reed-Tsochas, and R. I. Dunbar, “Persistence of social signatures in human communication,” Proceedings of the National Academy of Sciences, vol. 111, no. 3, pp. 942–947, 2014.
  * (13) S. Centellegher, E. López, J. Saramäki, and B. Lepri, “Personality traits and ego-network dynamics,” PLOS ONE, vol. 12, pp. 1–17, 03 2017.
  * (14) T. A. Vlahovic, S. Roberts, and R. Dunbar, “Effects of duration and laughter on subjective happiness within different modes of communication,” Journal of Computer-Mediated Communication, vol. 17, pp. 436–450, 2012.
  * (15) D. J. Reid and F. Reid, “Textmates and text circles: Insights into the social ecology of SMS text messaging,” in Mobile World. Computer Supported Cooperative Work (L. Hamill, L. A, and D. D, eds.), London: Springer, 2005.
  * (16) A. A. Nanavati, R. Singh, D. Chakraborty, K. Dasgupta, S. Mukherjea, G. Das, S. Gurumurthy, and A. Joshi, “Analyzing the structure and evolution of massive telecom graphs,” IEEE Transactions on Knowledge and Data Engineering, vol. 20, no. 5, pp. 703–718, 2008.
  * (17) Y. Wang, M. Faloutsos, and H. Zang, “On the usage patterns of multimodal communication: Countries and evolution,” in 2013 Proceedings IEEE INFOCOM, pp. 3135–3140, 2013.
  * (18) M. Zignani, C. Quadri, S. Bernardinello, S. Gaito, and G. P. Rossi, “Calling and texting: Social interactions in a multidimensional telecom graph,” in 2014 Tenth International Conference on Signal-Image Technology and Internet-Based Systems, pp. 408–415, 2014.
  * (19) C. Quadri, M. Zignani, L. Capra, S. Gaito, and G. P. Rossi, “Multidimensional human dynamics in mobile phone communications,” PLOS ONE, vol. 9, no. 7, pp. 1–12, 2014.
  * (20) J. Saramäki and E. Moro, “From seconds to months: an overview of multi-scale dynamics of mobile telephone calls,” The European Physical Journal B: Condensed Matter and Complex Systems, vol. 88, pp. 1–10, 2015.
  * (21) M. Karsai, M. Kivelä, R. K. Pan, K. Kaski, J. Kertész, A.-L. Barabási, and J. Saramäki, “Small but slow world: How network topology and burstiness slow down spreading,” Phys. Rev. E, vol. 83, p. 025102, 2011\.
  * (22) M. Kivelä, R. K. Pan, K. Kaski, J. Kertész, J. Saramäki, and M. Karsai, “Multiscale analysis of spreading in a large communication network,” J. Stat. Mech. Theor. Exp., vol. 2012, no. 03, p. P03005, 2012\.
  * (23) S. G. B. Roberts and R. I. M. Dunbar, “The costs of family and friends: an 18-month longitudinal study of relationship maintenance and decay,” Evolution and Human Behavior, vol. 32, pp. 186–197, 2011.
  * (24) V.-P. Backlund, J. Saramäki, and R. K. Pan, “Effects of temporal correlations on cascades: Threshold models on temporal networks,” Phys. Rev. E, vol. 89, p. 062815, 2014.
  * (25) J. Lin, “Divergence measures based on the shannon entropy,” IEEE Transactions on Information theory, vol. 37, no. 1, pp. 145–151, 1991.
  * (26) R. Dunbar, “The anatomy of friendship,” Trends in cognitive sciences, vol. 22, no. 1, pp. 32–51, 2018.
  * (27) J. Wiese, J.-K. Min, J. I. Hong, and J. Zimmerman, “You never call, you never write: Call and SMS logs do not always indicate tie strength,” in Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, pp. 765–774, ACM, 2015.
  * (28) A. Godoy-Lorite, R. Guimerà, and M. Sales-Pardo, “Long-term evolution of email networks: statistical regularities, predictability and stability of social behaviors,” PloS one, vol. 11, no. 1, p. e0146113, 2016.
  * (29) A. Caspi, B. W. Roberts, and R. L. Shiner, “Personality development: Stability and change,” Annu. Rev. Psychol., vol. 56, pp. 453–484, 2005.
  * (30) K. Bhattacharya, A. Ghosh, D. Monsivais, R. I. Dunbar, and K. Kaski, “Sex differences in social focus across the life cycle in humans,” Royal Society open science, vol. 3, no. 4, p. 160097, 2016.
  * (31) S. B. Roberts and R. I. Dunbar, “Managing relationship decay,” Human Nature, vol. 26, no. 4, pp. 426–450, 2015.
  * (32) R. M. Milardo, M. P. Johnson, and T. L. Huston, “Developing close relationships: Changing patterns of interaction between pair members and social networks.,” Journal of Personality and Social Psychology, vol. 44, no. 5, p. 964, 1983.


# Transformer-Guided Convolutional Neural Network for Cross-View
Geolocalization

Teng Wang  School of Automation
Southeast University
<EMAIL_ADDRESS>Shujuan Fan  School of Automation
Southeast University
<EMAIL_ADDRESS>Daikun Liu  School of Automation
Southeast University
<EMAIL_ADDRESS>Changyin Sun  School of Automation
Southeast University
<EMAIL_ADDRESS>
###### Abstract

Ground-to-aerial geolocalization refers to localizing a ground-level query
image by matching it to a reference database of geo-tagged aerial imagery.
This is very challenging due to the huge perspective differences in visual
appearances and geometric configurations between these two views. In this
work, we propose a novel Transformer-guided convolutional neural network
(TransGCNN) architecture, which couples CNN-based local features with
Transformer-based global representations for enhanced representation learning.
Specifically, our TransGCNN consists of a CNN backbone extracting feature map
from an input image and a Transformer head modeling global context from the
CNN map. In particular, our Transformer head acts as a spatial-aware
importance generator to select salient CNN features as the final feature
representation. Such a coupling procedure allows us to leverage a lightweight
Transformer network to greatly enhance the discriminative capability of the
embedded features. Furthermore, we design a dual-branch Transformer head
network to combine image features from multi-scale windows in order to improve
details of the global feature representation. Extensive experiments on popular
benchmark datasets demonstrate that our model achieves top-1 accuracy of
94.12% and 84.92% on CVUSA and CVACT_val, respectively, which outperforms the
second-performing baseline with less than 50% parameters and almost
$\mathbf{2\times}$ higher frame rate, therefore achieving a preferable
accuracy-efficiency tradeoff.

###### Index Terms:

cross-view geolocalization, convolutional neural network, multi-scale window
Transformer, attention mechanism

##  I Introduction

Ground-to-aerial cross-view geolocalization aims to determine the geographical
location of a query ground image by matching it with a large set of geo-
referenced database aerial/satellite images, as illustrated in Fig. 1. It
plays an important role in robot navigation [1], autonomous driving [2] and
augmented reality [3]. Although it has broad applications, this task is
extremely challenging due to the vast differences in both visual appearances
and spatial configurations between the two views. As a consequence, learning
effective feature representations for these two views can be very difficult
and has thus attracted great attention of researchers.

Figure 1: Illustration of ground-to-aerial cross-view geolocalization task.

In the past few years, convolutional neural networks (CNNs) have significantly
advanced the task of ground-to-aerial geolocalization. Prior works [4, 5]
extract features directly using the CNNs pre-trained for image classification.
Later on, [6, 7, 8] construct different Siamese CNNs to learn a more universal
cross-view feature representation from corresponding datasets. Several more
recent works incorporate CNNs with NetVlad layers [9], capsule networks [10]
or attention mechanisms [11, 12] to further enhance discriminative feature
representation learning. These CNN architectures typically leverage a stack of
convolutional layers to increasingly enlarge receptive fields and collect
local features in a hierarchical fashion as powerful feature representations.
Despite the great improvements, these CNN-based models experience difficulty
in capturing long-range dependency, and therefore fail to make more accurate
predictions in complex scenarios, where visual signals are ambiguous or
incomplete.

Recently, Natural Language Processing (NLP) field has witnessed the rapid rise
of Transformer with self-attention [13] in various powerful language modeling
architectures, which model long-range interaction in a scalable and parameter-
free manner. The success of Transformer on many NLP tasks strongly motivates
researchers to push the limits of computer vision tasks by integrating CNN-
based architectures with Transformer-style modules. With regard to cross-view
geolocalization, Yang et al. [14] make the first attempt to introduce
Transformer and propose a novel structure, termed EgoTR, which consists of a
ResNet backbone extracting CNN feature map from an input image and a Vision
Transformer (ViT) [15] modeling global context from the CNN map. Such a
Transformer-based model shows superiority over the dominant CNN-based
counterparts. However, this structure requires a large amount of parameters
and cannot meet real-time processing requirement, both of which are practical
constraints and hamper its widespread adoption.

In this paper, we hypothesize that Transformer could be strategically
introduced into the CNN structure to improve accuracy of ground-to-aerial
geolocalization, while concurrently maintaining a high degree of memory
efficiency and frame rate. To verify our hypothesis, we propose a novel
network structure, called Transformer-Guided Convolutional Neural Network
(TransGCNN), which couples CNN-based local features with Transformer-based
global representations for enhanced representation learning. As shown in Fig.
2, TransGCNN consists of a CNN backbone and a Transformer head. The CNN
backbone follows the design of VGG16 [16] to extract local features, whereas
the Transformer head operates on CNN feature map to model global context.
Different from prior works, our Transformer head acts as a spatial attention
module to select salient CNN features as the final feature representation.
Such a coupling procedure allows us to leverage a lightweight Transformer
network to greatly enhance the discriminative capability of the embedded
features. Furthermore, considering that vanilla ViT block performs self-
attention over the whole input map and easily miss local details, we design a
dual-branch vision Transformer to combine image features from multi-scale
windows to produce stronger features. Specifically, at a global-level, we
perform self-attention over the whole input feature map; whereas at the part-
level, we split the input feature map into several non-overlapping vertical
parts and perform self-attention over each part separately. The enhanced
feature maps from multi-scale windows are fused dynamically to complement each
other.

We conduct extensive experiments on two popular benchmarks, CVUSA [17] and
CVACT_val [18]. Experiment results demonstrate that our TransGCNN achieves
state-of-the-art performance, while being lightweight and fast. It outperforms
prior CNN-based models by a large margin in recalls. On the other hand, it
achieves competitive recalls while utilizing fewer parameters and attaining
higher frame rate compared to prior Transformer-based model. Main
contributions of our work are summarized as follows:

  * •

We design a hybrid network structure, which leverages a Transformer-style head
network as the spatial attention module to select salient CNN features as the
final image descriptor. Such a coupling mechanism enables our model to improve
the discriminative capability of the embedded features, while maintaining
lightweight and fast.

  * •

We propose a novel multi-scale window Transformer by performing self-attention
over multi-scale windows of the input map separately and fusing multi-scale
information dynamically. This module has proved to enhance details of the
global representation.

  * •

Extensive experiments on benchmark datasets confirm that our model achieves a
preferable trade-off between accuracy and efficiency. It achieves competitive
accuracy as the second-performing EgoTR with less than 50% parameters and
about $\mathbf{2\times}$ higher frame rate.

The rest of this paper is organized as follows. In Section II, we provide an
overview of related literature in cross-view geolocalization. Section III
gives a detailed description of our proposed approach. In Section IV, we
evaluate the performance of our approach by conducting extensive experiments
for ablation studies and comparing it to the state-of-the-art methods. We draw
conclusions in Section V.

Figure 2: Overview of our proposed Transformer-guided convolutional neural
network architecture for cross-view geolocalization.

##  II Related Work

###  II-A Traditional approaches

In the early stage, handcrafted features are widely used to address the tasks
of image matching and image retrieval. Hays et al. [19] design a variety of
features by expert knowledge of designers, including color histogram, texton
histograms, line features, gist descriptors, geometric context, and employ the
$\ell_{2}$ distance between the above feature descriptors as the distance
metric to search over a large-scale image database. Lin et al. [20] represent
each image using four features, including HoG [21], self-similarity [22], gist
[23], and color histograms, and proposes two data-driven approaches to achieve
cross-view geolocalization. In recent years, SIFT [24], SURF [25], ORB [26]
receive much attention due to the fact that they are invariant to rotation,
scale scaling and brightness change, and also stable to a certain extent for
angle change, affine transformation and noise, and are widely used in regular
image matching task [27, 28, 29, 30]. Inspired by the success of these
features descriptors in regular image matching, [31, 32, 33] explore SIFT and
SURF descriptors to match ground images against aerial images for cross-view
geolocalization. However, these handcrafted features are less robust to
lighting variations, viewpoint changes and other noises, thus limiting the use
of these approaches in real applications to some extent.

###  II-B Deep learning based approaches

Inspired by the great success of CNNs in many computer vision tasks, Workman
and Jacobs [4, 5] first introduce CNNs into the field of cross-view image
retrieval, and employ the CNNs pre-trained for classification to extract deep
features. The above works indicate better discriminativeness of deep features
compared to handcrafted features. However, these approaches directly utilize
the generic feature representations designed for object classification,
resulting in relatively low image retrieval accuracy. To that end, Lin et al.
[6] construct a two-branch Siamese network to map two views into a common
feature embedding space, and a weighted soft-margin triplet loss is utilized
to speed up the network learning; VO and Hays [7] explore several deep CNN
architectures including Classification, Hybrid, Siamese and Triplet networks
for cross-view matching, and experimental results prove that triplet network
outperforms other CNN architectures by a large margin. Recent works put more
emphasis on designing CNN architectures for better representation learning. Hu
et al. [9] incorporate a learnable NetVLAD layer [34] into a two-branch
Siamese CNN architecture to learn view-invariant feature representations. Sun
et al. [10] couple the powerful ResNet [35] backbone network with capsule
layers [36] to model high-level semantics. Cai et al. [11] propose a
lightweight attention module by combining spatial and channel attention
mechanisms to emphasis visually salient features in input images. Shi et al.
[12] design a multi-head spatial attention module to alleviate content-
dependent domain discrepancy. Despite great improvements, few of the above
works consider the long-distance relationships among visual elements, which
decreases the discriminative capability of the embedded features. To tackle
the challenge, Yang et al. [14] make the first exploration to introduce
transformer and propose a hybrid structure consisting of a ResNet backbone
extracting CNN feature map from an input image and a ViT modeling global
context from the CNN map. However, this hybrid structure requires a large
amount of parameters, and suffers from low frame rate. Different from existing
methods, we design a novel network structure, which leverages multi-scale
window Transformer as the spatial attention module to select salient CNN
features as the final image descriptor. We demonstrate that our method
achieves a preferable trade-off between accuracy and efficiency.

##  III The Proposed Method

###  III-A Overview

In this work, the ground-to-aerial geolocalization problem is regarded as a
standard image retrieval task. We follow [12] to employ a two-step approach to
solve it. The first step is to apply a regular polar transform to warp aerial
images in order to build more apparent spatial correspondences between aerial
and ground view images. Then, we adopt a domain-specific Siamese-like
architecture with two independent branches of the same architecture to
separately learn representations for these two views. A novel Transformer-
guided convolutional neural network, termed TransGCNN, is proposed for each
branch. The design of TransGCNN draws inspiration from how human beings
observe the world. Given a scene, humans typically first glance at the global
view and then gaze into salient local details under the guidances of the
global context to recognize the scene. Motivated by this, we propose
TransGCNN, which consists of a CNN backbone, where the first thirteen layers
of VGG16 [16] are adopted to capture the local patterns, and a multi-scale
window Transformer head, where self-attention is performed over multi-scale
windows of the CNN feature map to capture the global representation, as shown
in Fig. 2. Given local and global features, we feed them into the feature
coupling unit (FCU), which innovatively treats each channel of the global
feature map as one spatial attention map to re-weight the CNN feature map for
emphasizing salient local features. We concatenate the embedded features from
multiple attention channels together as our final image descriptor. During
training, the weighted triplet loss is utilized to supervise the whole deep
model.

###  III-B Polar Transform

Based on the fact that pixels lying on the same azimuth direction in an aerial
image approximately correspond to a vertical image column in the ground view
image, we explicitly apply a regular polar transform to warp the aerial images
and thus to some extent bridge the geometric correspondence gap between these
two different view domains. Specifically, we take the center of each aerial
image as the polar origin and the north direction as angle $0^{\circ}$ in the
polar transform. Suppose the sizes of the original ground and aerial images
are $W_{g}\times H_{g}$ and $W_{a}\times H_{a}$, respectively. Given any point
$({x^{s}},{y^{s}})$ in an original aerial image, its new location
$({x^{t}},{y^{t}})$ in the target transformed aerial image could be computed
as follows.

$\begin{split}{x^{t}}=\frac{W_{a}}{2}+\frac{A}{2}\frac{{{y^{s}}}}{H_{g}}\sin(\frac{{2\pi}}{W_{g}}{x^{s}})\\\
{y^{t}}=\frac{H_{a}}{2}-\frac{A}{2}\frac{{{y^{s}}}}{H_{g}}\cos(\frac{{2\pi}}{W_{g}}{x^{s}})\end{split}$
(1)

After polar transform, the objects in the transformed aerial images will lie
in similar positions to their counterparts in the ground images, as shown in
Fig. 2. Note that the size of the transformed satellite images is constrained
to be the same as that of the ground ones in order to facilitate training of
our two-branch Siamese network. More details about the polar transform could
be found in [12].

###  III-C Multi-Scale Window Transformer

Our multi-scale window Transformer consists of two parallel branches: a global
Transformer and a part-level Transformer. Both branches receives the CNN
feature map $F_{a}$ as inputs. Here, $F_{a}$ is obtained by performing
$1\times 1$ convolution on the CNN feature map $F$ to adjust its channel
dimension, such that pre-trained parameters on ImageNet could be used to
initialize these two Transformers. The enhanced feature maps from these two
branches are fused to derive the final global representation.

Global Transformer. This module contains $L$ repeated vision Transformer
blocks [15]. Each block consists of a single-head self-attention (SHSA) module
and a multi-layer perceptron (MLP) block. Layer normalization (LN) is applied
before each layer and residual connection. To capture global context, we feed
the CNN feature map $F_{a}\in\mathbb{R}^{H\times W\times C}$ into the
$L$-layer Transformer encoder by reshaping it into a sequence of tokens
$F_{t}\in\mathbb{R}^{HW\times C}$. Each Transformer block processes input
features as follows:

$\mathbf{z}_{\ell}^{\prime}=\text{SHSA}(\text{LN}(\mathbf{z}_{\ell-1}))+\mathbf{z}_{\ell-1},$
(2)
$\mathbf{z}_{\ell}=\text{MLP}(\text{LN}(\mathbf{z}_{\ell}^{\prime}))+\mathbf{z}_{\ell}^{\prime},$
(3)

where $\mathbf{z}_{\ell}\in\mathbb{R}^{HW\times C},\ell\in\\{1,\cdots,L\\}$,
is the encoded image representations at the $\ell$-th block, and we set
$\mathbf{z}_{\ell-1}=F_{t}$ for $\ell=1$. SHSA enables the Transformer to
model a global relationship in a parameter-free manner, which is formulated
as:

$\text{SHSA}(X)=\text{Softmax}(\frac{QK^{T}}{\sqrt{D}})V.\hskip 11.38092pt$
(4)

Here, $Q$, $K$, $V\in\mathbb{R}^{N\times D}$ are respectively the $query$,
$key$ and $value$ matrices, which are linear projections of input
$X\in\mathbb{R}^{N\times D}$, $N$ is the length of input token sequence, and
$D$ is the token embedding size. More details about vision Transformer block
could be found in [15].

Part-level Transformer. Although the global Transformer could effectively
capture long-range dependency, it easily misses local details. To this end, we
introduce a part-level Transformer module to enhance local details of the
global representation. This module is composed of $L$ part-level Transformer
blocks, whose overall architecture is illustrated in Fig. 3. The main
difference between our proposed part-level Transformer block and the regular
one is that we split the input feature map into multiple non-overlapping
partitions and perform self-attention with shared parameters over each
partition separately. As different scene parts along horizontal direction
typically carry different semantics, we split the input feature map into $w$
vertical partitions. This process could be described as follows:

$\mathbf{\hat{z}}_{\ell-1}=[\mathbf{\hat{z}}_{\ell-1}^{1,1},\mathbf{\hat{z}}_{\ell-1}^{1,2},\cdots,\mathbf{\hat{z}}_{\ell-1}^{H,W}],$
(5)

where $\mathbf{\hat{z}}_{\ell-1}\in\mathbb{R}^{HW\times C}$ is the encoded
image representations at the $\ell-1$-th block, and we set
$\mathbf{\hat{z}}_{\ell-1}=F_{t}$ for $\ell=1$.
$\mathbf{\hat{z}}_{\ell-1}^{i,j}$ is the feature token at position $(i,j)$ if
reshaping the token sequence $\mathbf{\hat{z}}_{\ell-1}$ back into a 2D
feature map. To capture local details, $\mathbf{\hat{z}}_{\ell-1}$ is first
split into $w$ partitions before being input into the $\ell$-th block:

$\mathbf{\hat{z}}_{\ell-1}=[\mathbf{\hat{z}}_{\ell-1}^{1},\mathbf{\hat{z}}_{\ell-1}^{2},\cdots,\mathbf{\hat{z}}_{\ell-1}^{w}]$
(6)

where

$\mathbf{\hat{z}}_{\ell-1}^{j}=[\mathbf{\hat{z}}_{\ell-1}^{1,I_{j}+1},\cdots,\mathbf{\hat{z}}_{\ell-1}^{1,I_{j}+M},\cdots,\mathbf{\hat{z}}_{\ell-1}^{H,I_{j}+1},\cdots,\mathbf{\hat{z}}_{\ell-1}^{H,I_{j}+M}]$
(7)

where $M=\frac{W}{w}$, and $I_{j}=M*(j-1)$. Afterwards, we apply SHSA to each
partition $\mathbf{z}_{\ell-1}^{j}\in\mathbb{R}^{HM\times C}$:

$\mathbf{\hat{z}}_{\ell}^{\prime,j}=\text{SHSA}(\text{LN}(\mathbf{\hat{z}}_{\ell-1}^{j}))+\mathbf{\hat{z}}_{\ell-1}^{j}.$
(8)

And then, all partitions are merged back into one feature map and go through
the remaining MLP module to fuse information across channels for each point
separately:

$\mathbf{\hat{z}}_{\ell}^{\prime}=\text{Merge}(\mathbf{\hat{z}}_{\ell}^{\prime,1},\mathbf{\hat{z}}_{\ell}^{\prime,2},\cdots,\mathbf{\hat{z}}_{\ell}^{\prime,w})$
(9)
$\mathbf{\hat{z}}_{\ell}=\text{MLP}(\text{LN}(\mathbf{\hat{z}}_{\ell}^{\prime}))+\mathbf{\hat{z}}_{\ell}^{\prime}.$
(10)

where Merge is the operation to re-arrange points in each partition back in
their original orders in the feature map. Such Split-Merge designs give part-
level Transformer the advantage of retaining details while modeling long-range
dependency.

Figure 3: An illustration of our part-level Transformer block.

(a) CVUSA

(b) CVACT

Figure 4: Ground-to-aerial image pairs sampled from CVUSA and CVACT. Each
subfigure illustrate a ground image (left) and an aerial image (right)

The output feature maps $G=\mathbf{z}_{L}$ and $P=\mathbf{\hat{z}}_{L}$
respectively from the global and part-level Transformers are fused by the
element-wise summation followed by one SE (i.e., Squeeze-and-Excitation) block
[37] to derive the final global representation $M$. Here, the SE block is
introduced to capture channel-wise dependency and emphasize informative
channels. It is worth mentioning that we do not utilize position encodings in
both global and part-level Transformers. This is because the global feature
map $M$ is just treated as the spatial attention map to select salient CNN
features as the final embedded features, and the CNN features already carry
abundant position information.

###  III-D Feature Coupling Unit

Given the local features $F$ and global representation $M$ respectively from
the CNN backbone and Transformer head, how to utilize them to effectively
enhance the discriminative capability of the embedded features is an important
issue. To solve, we propose an attention-based feature coupling unit, where
the global feature map $M$ is treated as the spatial attention map to guide
the model to focus on salient features on the feature map $F$. Furthermore,
motivated by the feature aggregation strategy [12], we aim to improve the
robustness of our feature representations by aggregating multiple embedded
features. Towards this goal, we feed the global feature map $M$ into a
$1\times 1$ convolution layer to adjust its channel dimension and treat each
channel as one specific spatial attention map to generate the embedded
features. The attention-based feature coupling procedure could be described as
follows.

$A=\mathbf{Conv_{1\times 1}}(M)$ (11) $f^{c}_{i}=<F^{c},A^{i}>$ (12)

where $F^{c}$ represents the CNN feature map in the $c$-th channel, $A^{i}$
denotes the attention map in the $i$-th channel, $<\cdot,\cdot>$ denote the
Frobenius product of the two inputs, and $f^{c}_{i}$ is the embedded feature
activation for the $c$-th channel from the $i$-th attention branch.

We concatenate the embedded features from multiple attention channels together
as our final image descriptor f.

$\textbf{f}=\\{f^{1}_{1},\cdots,f^{C}_{1},f^{1}_{2},\cdots,f^{C}_{2},\cdots,f^{1}_{K},\cdots,f^{C}_{K}\\}$
(13)

where $C$ and $K$ represent channel dimensions of the CNN feature map $F$ and
attention map $A$, respectively.

###  III-E Loss Function

Recently, triplet loss is widely used in cross-view image matching and
retrieval tasks [12, 9, 18], and has proved to achieve impressive
performances. It aims to minimize the distance between matched pairs while
maximizing the distance between unmatched pairs in the embedding space.
Similar to [12], we employ a weighted soft-margin triplet loss to supervise
the whole model. Specifically, given an image triplet $\left\langle
I_{a},I_{p},I_{n}\right\rangle$, the retrieval loss $L$ is defined as follows:

$\begin{split}L=\log(1+{e^{{\rm{\gamma}}({d_{f}^{pos}}-{d_{f}^{neg}})}}),\end{split}$
(14)

where $d_{f}^{pos}$ and $d_{f}^{neg}$ refer to the $\ell_{2}$ distance between
the feature representations from matched $\left\langle
I_{a},I_{p}\right\rangle$ and unmatched $\left\langle
I_{a},I_{n}\right\rangle$ image pairs, respectively; $\gamma$ is a parameter
to adjust the gradient of the loss, and is empirically set to 10.

##  IV Experiments

###  IV-A Experiment Details

Datasets. We conduct all our experiments on two standard benchmark datasets:
CVUSA [17] and CVACT [18]. Each dataset provides a total of 35,532 ground-to-
aerial image pairs for training. CVUSA provides a total of 8,884 image pairs
for testing and CVACT provides the same number of image pairs for validation
(denoted as CVACT_val). Additionally, CVACT also provides a total of 92,802
image pairs with accurate geo-tags to support fine-grained city-scale geo-
localization (denoted as CVACT_test). In CVACT_test, all aerial images within
5 meters away from a query ground image are regarded as the ground-truth
correspondences for the query image. That means there may exist several
corresponding aerial images for each query ground image in this dataset. In
this work, we follow previous works [12, 38] to employ CVUSA and CVACT_val to
evaluate the performance of our proposed method on standard cross-view
geolocalization task. In both datasets, each image pair consist of a panoramic
street-view image and its ground-truth satellite image, which are both north-
aligned. Note that the street-view and aerial images in both datasets are
captured at different times, making ground-to-aerial geolocalization more
challenging. Sampled image pairs from these two datasets are presented in Fig.
4.

TABLE I: Performance comparison on recall accuracy with the state-of-the-art on CVUSA & CVACT_val dataset. Model | CVUSA | CVACT_val
---|---|---
r@1 |  r@5 |  r@10 |  r@1% |  r@1 |  r@5 |  r@10 |  r@1%
CVM-NET [9] | 22.53 | 50.01 | 63.19 | 93.62 | 20.15 | 45.00 | 56.87. | 87.57
Liu and Li [18] | 40.79 | 66.82 | 76.36 | 96.08 | 46.96 | 68.28 | 75.48 | 92.01
Regmi and Shah [39] | 48.75 | - | 81.27 | 95.98 | - | - | - | -
CVFT [40] | 61.43 | 84.69 | 90.49 | 99.02 | 61.05 | 81.33 | 86.52 | 95.93
SAFA [12] | 89.84 | 96.93 | 98.14 | 99.64 | 81.03 | 92.80 | 94.84 | 98.17
Shi et al. [41] | 91.93 | 97.50 | 98.54 | 99.67 | 82.49 | 92.44 | 93.99 | 97.32
EgoTR [14] | 93.69 | 98.15 | 98.90 | 99.74 | 84.19 | 94.37 | 95.85 | 98.26
TransGCNN (ours) | 94.15 | 98.21 | 98.94 | 99.79 | 84.92 | 94.46 | 95.88 | 98.36

Implementation Details. We resize both the panoramic street-view image and
polar-transformed satellite image into $128\times 512$, and perform a mean
normalization on resized images before feeding them into our proposed
TransGCNN. The first 13 layers of VGG16 with pretrained weights on ImageNet is
employed as our CNN backbone to extract local features. The output CNN feature
map is fed into a $1\times 1$ convolutional layer to increase its channel
dimension to 768. The enhanced CNN feature map goes through the multi-scale
window transformer module to capture the global representation. For part-level
Transformer, we divide the input CNN feature map into 5 vertical partitions
and perform self-attention over each part separately. We empirically set the
depth $L$ of both global and part-level Transformer to 2 and initialize them
with pretrained parameters on ImageNet. The channel dimension $K$ of the
attention map is set to $8$. Our experiments are based on Pytorch. The models
are trained using AdamW optimizer with an initial learning rate set to
$10^{-5}$. If not specified, all the experiments are conducted on a 32GB
NVIDIA GeForce RTX 3090 GPU, and the batch size $B$ is set to 32. We utilize
exhaustive mini-batch strategy [7] to generate image triplets within a batch.
For each ground-view image in a mini-batch, there are 1 matching aerial images
and $B-1$ unmatched aerial images, generating a total of $B(B-1)$ triplets.
Similarly, there are 1 matching ground-view images and $B-1$ unmatched ground-
view images for each satellite image, creating $B(B-1)$ triplets in total.
Thus, we have a total of $2B(B-1)$ image triplets within each mini-batch.

Evaluation Metrics. We adopt the recall accuracy at top K as the metric to
evaluate the ground-to-aerial image retrieval accuracy of our deep model. In
this work, we follow [12] to consider the recall accuracy at top-1, top-5,
top-10, up to top-1% (i.e., r@1, r@5, r@10, r@1%). Besides, we count the
number of parameters and frames per second (FPS) to measure model size and
processing speed of our method, respectively.

###  IV-B Comparison to State-of-the-Art Models

In this section, we conduct extensive evaluations on the standard benchmarks
CVUSA [17] and CVACT_val [18] by comparing our TransGCNN with several recent
state-of-the-art methods: CVM-Net [9], Liu and Li [18], CVFT[40], SAFA [12],
Shi et al. [41], and EgoTR [14]. Among these six baseline models, the first
five models employ stacked convolutional layers to collect local features as
the final feature representation, while the last EgoTR introduces vision
Transformer ViT to capture the global representation. It is worth mentioning
that our TransGCNN shares similar structure with CNN-based SAFA, except that
we leverage multi-scale window Transformer to design the spatial attention
module for emphasizing salient local features. For fair performance
comparison, we compute the recall accuracies of these existing approaches
using the released models or codes provided by the authors.

(a) Learning curves

(b) Recalls

Figure 5: Comparison of learning curves and recalls of different models on
CVUSA datasets.

Accuracy. We evaluate our TransGCNN on standard cross-view geolocalization.
Table I reports recalls at top-1, top-5, top-10, up to top 1% on CVUSA and
CVCAT_val datasets. From the experiment results, we observe that our TransGCNN
significantly surpasses those CNN-based baselines in the most challenging r@1
accuracy, indicating better discriminative capability of the feature
representation learned from our TransGCNN. In particular, our TransGCNN
achieves r@1 of 94.15% on CVUSA dataset compared to 89.84% obtained by SAFA,
and the TransGCNN outperforms SAFA by a significant margin of 3.89% points at
r@1 on CVACT_val dataset. The superiority of our proposed TransGCNN over CNN-
based SAFA suggests the effectiveness of leveraging Transformer-style module
to implement the spatial attention mechanism. This is because Transformer
excels at capturing global contexts, which are typically critical for
recognizing discriminative parts. In addition, our TransGCNN performs slightly
better than the Transformer-based EgoTR on both datasets (by 0.43% on CVUSA
and by 0.73% on CVACT_val). In summary, our TransCGNN achieves 94.15% and
84.92% top-1 accuracy on CVUSA and CVCAT_val datasets, respectively, which are
new state-of-the-art performances.

Model Size and Processing Speed. Real-world ground-to-aerial geolocalization
applications also pose high requirements on model size and processing speed of
the cross-view image retrieval network. Thus, we count the amount of
parameters for different models, and test these models with identical testing
settings to record their FPS. Specifically, we run all these models on a RTX
3090, 5 Core Intel(R) Xeon(R) Silver 4210R<EMAIL_ADDRESS>The results are listed
in Table II. We observe that our TransGCNN requires $2.23\times$ fewer
parameters and attains almost $2\times$ higher FPS compared to EgoTR, while
achieving higher recalls on both datasets. This observation could be explained
as follows. Our TransGCNN employ only 2 Transformer blocks in both global and
part-level Transformers to design the spatial attention module, however EgoTR
stacks 12 Transformer blocks to design the backbone network, which greatly
increases the number of parameters and processing time per frame. In addition,
compared to CNN-based SAFA, our TransGCNN improve r@1 accuracy by a
significant margin (i.e., 4.31% and 3.89% respectively for CVUSA and
CVACT_Val) at the cost of requiring $2.97\times$ more parameters and obtaining
$1.55\times$ lower FPS. The above two observations demonstrate that our
TransGCNN significantly improves the recall accuracy of dominant CNN-based
solutions, while maintaining a relatively higher memory efficiency and frame
rate compared to the second-performing EgoTR. In other words, our TransGCNN
achieves a preferable trade-off between accuracy and efficiency.

TABLE II: Comparison on model size and FPS with the state-of-the-art Model |  #Param (M) |  #Frame Per Second
---|---|---
SAFA [12] | 29.50 | 35.90
Shi et al. [41] | 17.90 | 22.72
EgoTR [14] | 195.90 | 13.37
TransGCNN (ours) | 87.80 | 23.09

Training Cost. Our TransGCNN and the second-performing EgoTR both leverage
Transformer-style modules to enhance feature representation learning. However,
these two models utilize Transformer in different manners. Specifically, EgoTR
employs vision Transformer ViT as backbone to extract the global
representation, while our TransGCNN treats the multi-scale window Transformer
as the spatial attention module to select salient CNN features as the final
embedded features. In order to explore how the incorporation of Transformer-
style modules affects the learning process, we depict the learning curves of
the two models on CVUSA dataset in Fig. 5 (a). Here, the batch size of 32 is
selected to guarantee that EgoTR achieves the best performance. We observe
that both TransGCNN and EgoTR take about 150 epochs to converge. Furthermore,
we would also like to know the impacts of batch size on both models, as it is
an important hyper-parameter in deep models. In our practice, we reduce the
batch size from 32 down to 16 and 8, and re-train both models on CVUSA
dataset. The learning curves of both models with different batch sizes are
depicted in Fig. 5 (a). Besides, the recall curves of the learned models with
different batch sizes on CVUSA dataset are given in Fig. 5 (b). We note some
particular results from these curves. First, reducing batch size causes a
slight decrease in the converge speed of EgoTR. Second, EgoTR suffers from
obvious accuracy degradation when the batch size is reduced. In specific, the
r@1 drops from 93.69 to 91.02 when the batch size is reduced down to 16, and
from 93.69 to 86.49 when the batch size is reduced down to 8. In contrast, our
TransGCNN does not show obvious performance degradation when the batch size is
reduced. The above observations indicate that the performance of EgoTR is
susceptible to batch size, and poses a higher requirement on the video memory
of the computing machine compared to our TransGCNN.

###  IV-C Ablation Studies

Does Part-level Transformer Help? In TransGCNN, we design a novel multi-scale
window Transformer as the spatial attention module to select salient CNN
features. Our multi-scale window Transformer consists of a regular global
Transformer and a novel part-level Transformer. To show the contributions of
the part-level Transformer, we consider two scenarios: (a) TransGCNN-SB, which
removes part-level Transformer while increasing the depth of the global
Transformer from 2 to 4 in order to keep the amount of parameters the same as
the original multi-scale window Transformer; (b) TransGCNN-SS, which replaces
the part-level Transformer with the global Transformer for constructing a two-
branch single-scale window Transformer. We test the two modified models on
CVUSA dataset, and show their recalls in Table III. We notice obvious accuracy
degradation in r@1 (1.46% and 5.65% respectively for TransGCNN-SB and
TransGCNN-SS), demonstrating the effectiveness of the part-level Transformer
in enhancing feature representation learning. This is because the part-level
Transformer excels at capturing small-scale information by limiting the
calculation of self-attention to a local window. Such local information
complements the large-scale information from the global Transformer to obtain
more comprehensive information.

TABLE III: Ablation studies of TransGCNN on part-level Transformer Model | CVUSA
---|---
r@1 |  r@5 |  r@10 |  r@1%
TransGCNN | 94.15 | 98.21 | 98.94 | 99.19
TransGCNN-SB | 92.69 | 97.66 | 98.58 | 99.71
TransGCNN-SS | 88.50 | 96.81 | 98.13 | 99.71
w/o SE | 92.96 | 98.01 | 98.91 | 99.72

Does Channel Attention Matter? Given feature maps respectively from the global
and part-level Transformers, we first perform pixel-wise summation and then
apply SE-based channel attention to re-weigh different channels for achieving
multi-scale information fusion. We show the contributions of the channel
attention by simply removing the SE block from the multi-scale window
Transformer and testing the reduced model on CVUSA dataset. The results are
shown in Table III. We observe that with the help of channel attention,
TransGCNN improves by the margin of 1.19% at r@1 (from 92.96 to 94.15). This
observation demonstrates that this dynamic information fusing mechanism does
play an important role in our multi-scale window Transformer module. This
might be explained by the fact that the SE block helps emphasize informative
channels and thus reduce channel redundancy.

The Number of Vertical Parts $\bm{w}$. Intuitively, $w$ determines the
granularity of part-level representations. We investigate the effects of the
number of vertical parts $w$ on retrieval accuracy by increasing $w$ from 1,
3, 5 up to 7, and reporting the recall rates under different values of $w$ in
Table IV. The results indicate that as $w$ increases from 1 to 5, we obtain
better recall performances. Specifically, obvious improvements (2.25% and
1.45%) for r@1 are obtained when $w$ increases from 1 to 3, and from 3 to 5.
However, the retrieval accuracy does not always increase with $w$. When $w$
increases from 5 up to 7, parts are too small to be discriminative, leading to
accuracy degradation. That means, an over-increased $w$ actually compromises
the discriminative power of part-level representations. Therefore, we set $w$
to be 5 across all our experiments. This is reasonable and could be explained
by the fact that panoramic street-view images are captured when the car is
driving on the road and thus these panoramic images typically consist of five
semantic components, including three lawn scenes and two roads.

TABLE IV: Impact of the number of parts $w$ on recall accuracy  Number of Parts | CVUSA
---|---
r@1 |  r@5 |  r@10 |  r@1%
$w$ = 1 | 90.35 | 97.25 | 98.52 | 99.79
$w$ = 3 | 92.60 | 98.01 | 98.72 | 99.78
$w$ = 5 | 94.15 | 98.21 | 98.94 | 99.79
$w$= 7 | 92.52 | 98.00. | 98.77 | 99.77

The Number of Attention Channels K. In our work, multiple attention maps are
introduced to focus the deep model on salient CNN features, with different
attention map capturing different discriminative cues. We investigate the
effects of the number of attention channels K on retrieval accuracy by
increasing K from 1, 2, 4 up to 8, and report the recall rates under different
values of K in Table V. The results indicate that as K increases, we can
obtain better recall performances. Note that significant improvement
($7.37\%$) for r@1 is obtained when K increases from 1 to 2. However, when K
increases from 2 to 4 and from 4 to 8, we only obtain improvements in r@1
smaller than $1.5\%$. Therefore, we do not increase K to an even larger
number, and set K to be 8 across all experiments.

TABLE V: Impact of Attention Channels K on recall accuracy  #Attention Channels | CVUSA
---|---
r@1 |  r@5 |  r@10 |  r@1%
K = 1 | 84.42 | 95.38 | 97.25 | 99.53
K = 2 | 91.79 | 95.57 | 98.41 | 99.68
K = 4 | 92.75 | 98.04 | 98.75 | 99.77
K = 8 | 94.15 | 98.21 | 98.94 | 99.79

(a) Street-view ground images

(b) Global spatial attention maps

(c) Part-level spatial attention maps

(d) Spatial attention maps from our TransGCNN

(e) Spatial attention maps from the baseline SAFA

Figure 6: Visualizations of spatial attention maps from our TransGCNN and the
baseline SAFA.

Visualization of Spatial Attention Maps. We visualize the spatial attention
maps learned from the baseline SAFA and our TransGCNN in Fig. 6. In addition,
we also present the enhanced feature maps from both the global and part-level
Transformers to illustrate how our TransGCNN actually work. We make the
following two observations from these attention maps. First, our TransGCNN is
capable of identifying salient roads and buildings, and paying more attention
to these objects. In comparison, the baseline SAFA focuses more on
distinguishing foreground from background (i.e., sky) and fails to identify
the relative importances of different foreground objects. The advantages of
our TransGCNN over SAFA further demonstrate the importances of utilizing
Transformer-style modules to capture global contexts for implementing spatial
attention mechanism. Second, the global and part-level branches in our
proposed multi-scale window Transformer module learns different features,
which complement each other to obtain more abundant descriptions of the
scenes.

Figure 7: Image retrieval examples on CVUSA dataset [17]. From left to right:
street-view query image and the Top 1-5 retrieved satellite images. Red
borders indicate ground-truth satellite images.

Figure 8: Examples of negative scenarios, where the ground-truth satellite
image is not the nearest top-1 retrieved images according to Euclidean
distance. From left to right: street-view query image and the Top 1-5
retrieved satellite image. Red borders indicate ground-truth satellite images.

###  IV-D Presentation and Discussion

We randomly choose five panoramic street-view images from test dataset and
perform image retrieval using our TransGCNN. The top-5 retrieved satellite
images are presented in Fig. 7. We observe that our method could successfully
localize ground-truth satellite images even when there exist huge visual
differences between ground-to-aerial image pairs. In addition, we also present
some negative scenarios in Fig. 8, where the ground-truth satellite image is
not the nearest top-1 retrieved image according to Euclidean distance in the
feature embedding space. It is easily observed that these negative samples are
really hard samples, which are even difficult for human to identify.

##  V Conclusions

In this paper, we propose a novel Transformer-guided convolutional neural
network architecture to couples CNN-based local features with Transformer-
based global representations for enhanced representation learning.
Specifically, we design a multi-scale window Transformer and treat it as the
spatial attention module to select salient CNN features as the final embedded
features. In particular, the multi-scale window Transformer is achieved by
combining image features from multi-scale windows to produce stronger global
contexts. Extensive experiments on popular benchmark datasets demonstrate that
our model achieves top-1 accuracy of 94.12% and 84.92% on CVUSA and CVACT_val,
respectively, which outperforms the second-performing baseline with less than
50% parameters and almost $\mathbf{2\times}$ higher frame rate, thus achieving
a preferable accuracy-efficiency tradeoff. In future, we will explore to
utilize our proposed method to address the more challenging task of unaligned
ground-to-aerial geolocalization.

## Acknowledgment

This work was supported by in part by National Natural Science Foundation of
China under Grant 61921004, and in part by ZhiShan Scholar Program of
Southeast University.

## References

  * [1] T. Senlet and A. Elgammal, “Satellite image based precise robot localization on sidewalks,” in _IEEE International Conference on Robotics and Automation_. IEEE, 2012, pp. 2647–2653.
  * [2] C. McManus, W. Churchill, W. Maddern, A. D. Stewart, and P. Newman, “Shady dealings: Robust, long-term visual localisation using illumination invariance,” in _IEEE international conference on robotics and automation (ICRA)_. IEEE, 2014, pp. 901–906.
  * [3] S. Middelberg, T. Sattler, O. Untzelmann, and L. Kobbelt, “Scalable 6-dof localization on mobile devices,” in _European conference on computer vision_. Springer, 2014, pp. 268–283.
  * [4] S. Workman and N. Jacobs, “On the location dependence of convolutional neural network features,” in _IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2015, pp. 70–78.
  * [5] S. Workman, R. Souvenir, and N. Jacobs, “Wide-area image geolocalization with aerial reference imagery,” in _IEEE International Conference on Computer Vision_ , 2015, pp. 3961–3969.
  * [6] T.-Y. Lin, Y. Cui, S. Belongie, and J. Hays, “Learning deep representations for ground-to-aerial geolocalization,” in _IEEE conference on Computer Vision and Pattern Recognition_ , 2015, pp. 5007–5015.
  * [7] N. Vo and J. Hays, “Localizing and orienting street views using overhead imagery,” in _European conference on computer vision_. Springer, 2016, pp. 494–509.
  * [8] N. Vo, N. Jacobs, and J. Hays, “Revisiting im2gps in the deep learning era,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 2621–2630.
  * [9] S. Hu, M. Feng, R. M. Nguyen, and G. H. Lee, “Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7258–7267.
  * [10] B. Sun, C. Chen, Y. Zhu, and J. Jiang, “Geocapsnet: ground to aerial view image geo-localization using capsule network,” in _IEEE Conference on Multimedia and Expo (ICME)_ , 2019, pp. 742–747.
  * [11] S. Cai, Y. Guo, S. Khan, J. Hu, and G. Wen, “Ground-to-aerial image geo-localization with a hard exemplar reweighting triplet loss,” in _IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8391–8400.
  * [12] Y. Shi, L. Liu, X. Yu, and H. Li, “Spatial-aware feature aggregation for cross-view image based geo-localization,” in _33rd Conference on Neural Information and Processing Systems (NIPS)_ , 2019.
  * [13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomze, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , 2017.
  * [14] H. Yang, X. Lu, and Y. Zhu, “Cross-view geo-localization with evolving transformer,” in _35th Conference on Neural Information and Processing Systems (NIPS)_ , 2021.
  * [15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, and et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [16] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2015.
  * [17] M. Zhai, Z. Bessinger, S. Workman, and N. Jacobs, “Predicting ground-level scene layout from aerial imagery,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 867–875.
  * [18] L. Liu and H. Li, “Lending orientation to neural networks for cross-view geolocalization,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [19] J. Hays and A. A. Efros, “Im2gps: estimating geographic information from a single image,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–8.
  * [20] T.-Y. Lin, S. Belongie, and J. Hays, “Cross-view image geolocalization,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 891–898.
  * [21] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [22] E. Shechtman and M. Irani, “Matching local self-similarities across images and videos,” in _IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2007, pp. 1–8.
  * [23] A. Oliva and A. Torralba, “Modeling the shape of the scene: A holistic representation of the spatial envelope,” _International journal of computer vision_ , vol. 42, no. 3, pp. 145–175, 2001.
  * [24] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [25] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in _European Conference on Computer Vision_. Springer, 2006, pp. 404–417.
  * [26] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: An efficient alternative to sift or surf,” in _International Conference on Computer Vision_. IEEE, 2011, pp. 2564–2571.
  * [27] Y. Bastanlar, A. Temizel, and Y. Yardimci, “Improved sift matching for image pairs with scale difference,” _Electronics Letters_ , vol. 46, no. 5, pp. 346–348, 2010.
  * [28] Z. Huijuan and H. Qiong, “Fast image matching based-on improved surf algorithm,” in _International conference on electronics, communications and control (ICECC)_. IEEE, 2011, pp. 1460–1463.
  * [29] A. Sedaghat and H. Ebadi, “Remote sensing image matching based on adaptive binning sift descriptor,” _IEEE transactions on geoscience and remote sensing_ , vol. 53, no. 10, pp. 5283–5293, 2015.
  * [30] E. Karami, S. Prasad, and M. Shehata, “Image matching using sift, surf, brief and orb: performance comparison for distorted images,” _arXiv preprint arXiv:1710.02726_ , 2017.
  * [31] S. Lefèvre, D. Tuia, J. D. Wegner, T. Produit, and A. S. Nassar, “Toward seamless multiview scene analysis from satellite to street level,” _Proceedings of the IEEE_ , vol. 105, no. 10, pp. 1884–1899, 2017.
  * [32] M. Noda, T. Takahashi, D. Deguchi, I. Ide, H. Murase, Y. Kojima, and T. Naito, “Vehicle ego-localization by matching in-vehicle camera images to an aerial image,” in _Asian Conference on Computer Vision_. Springer, 2010, pp. 163–173.
  * [33] A. Viswanathan, B. R. Pires, and D. Huber, “Vision based robot localization by ground to satellite matching in gps-denied situations,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 192–198.
  * [34] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in _IEEE conference on computer vision and pattern recognition_ , 2016, pp. 5297–5307.
  * [35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [36] S. Sabour, N. Frosst, and G. Hinton, “Dynamic routing between capsules,” in _IEEE Conference on Neural Information Processing Systems_ , 2017, pp. 3859–3869.
  * [37] J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu, “Squeeze-and-excitation network,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [38] T. Wang, Z. Zheng, C. Yan _et al._ , “Each part matters: Local patterns facilitate cross-view geo-localization,” in _arXiv:2008.11646_ , 2021.
  * [39] K. Regmi and M. Shah, “Bridging the domain gap for ground-to-aerial image matching,” in _IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 470–479.
  * [40] Y. Shi, X. Yu, L. Liu, T. Zhang, and H. Li, “Optimal feature transport for cross-view image geo-localization,” in _AAAI Conference on Artificial Intelligence_ , 2020, pp. 11 990–11 997.
  * [41] Y. Shi, X. Yu, D. Campbell, and H. Li, “Where am i looking at? joint location and orientation estimation by cross-view matching,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 4064–4072.