# View-Invariant Localization using Semantic Objects
in Changing Environments

Jacqueline Ankenbauer, Kaveh Fathian, Jonathan P. How J. Ankenbauer, K.
Fathian and J. P. How are with the Department of Aeronautics and Astronautics,
Massachusetts Institute of Technology. {jpedlow, kavehf<EMAIL_ADDRESS>Supported by The Boeing Company under Cooperative Agreement MRA#2017-656, ARL
DCIST under Cooperative Agreement W911NF-17-2-0181, and by UPenn under ONR
Award 584551.

###### Abstract

This paper proposes a novel framework for real-time localization and egomotion
tracking of a vehicle in a reference map. The core idea is to map the semantic
objects observed by the vehicle and register them to their corresponding
objects in the reference map. While several recent works have leveraged
semantic information for cross-view localization, the main contribution of
this work is a view-invariant formulation that makes the approach directly
applicable to any viewpoint configuration for which objects are detectable.
Another distinctive feature is robustness to changes in the
environment/objects due to a data association scheme suited for extreme
outlier regimes (e.g., 90% association outliers). To demonstrate our
framework, we consider an example of localizing a ground vehicle in a
reference object map using only cars as objects. While only a stereo camera is
used for the ground vehicle, we consider reference maps constructed a priori
from ground viewpoints using stereo cameras and Lidar scans, and georeferenced
aerial images captured at a different date to demonstrate the framework’s
robustness to different modalities, viewpoints, and environment changes.
Evaluations on the KITTI dataset show that over a $3.7$ km trajectory,
localization occurs in $36$ sec and is followed by real-time egomotion
tracking with an average position error of $8.5$ m in a Lidar reference map,
and on an aerial object map where $77\%$ of objects are outliers, localization
is achieved in $71$ sec with an average position error of $7.9$ m.

##  I INTRODUCTION

Localization is the process of determining the position and orientation (i.e.,
pose) of a vehicle in its environment. As global navigation satellite systems
can be denied (e.g., in subterranean environments), localization based on
onboard sensor observations of the surrounding environment is of great
interest. Given a reference map of the environment, the key challenge of
localization is to correctly associate vehicle observations to their
correspondence in the reference map. This difficulty can be caused by 1)
different perspectives/viewpoints between the reference and vehicle maps
(e.g., localizing a ground vehicle in an aerial map is challenging since the
same location is visually different from the ground and aerial views), 2)
changes in the environment (e.g., lighting differences, seasonal changes, and
objects being created, moved, or removed causes a discrepancy between
reference and vehicle maps), 3) different sensing modalities (e.g., localizing
a camera image in a Lidar point cloud is challenging due to different data
attributes).

Figure 1:  Localization and tracking the pose of a ground vehicle in a
reference object map (blue squares) built from cars in a georeferenced aerial
image. The ground vehicle is localized by creating an object map of the cars
observed while driving (red squares) and registering them with the cars in the
reference map. Over a $3.7$ km trajectory, our framework localizes the vehicle
after $476$ m (in $71$ sec), and tracks it henceforth with an average position
error of $7.9$ m. This is despite the ground vehicle dataset and the aerial
image being captured in different years and having nonidentical cars with only
$23\%$ (roughly) at the same location, and the reference map having an
accuracy of (at best) $2$ m.

Toward addressing the above three challenges, this work presents a novel
framework based on registering semantic objects observed by the vehicle
(vehicle map) to the objects in the environment (reference map). The proposed
framework is view-invariant because it uses a global registration technique
that can match objects in any configuration as long as objects are
observable/detectable in the vehicle and reference views. Additionally, the
chosen data association scheme makes the framework highly robust to
association outliers. Therefore, objects that are dynamic, missed,
misclassified, or represented poorly are treated as outliers and rejected–even
if they constitute a great portion of the data–without affecting the
localization. Further, using objects for localization introduces a degree of
robustness to environment appearance changes because object classifiers can be
trained to accurately identify objects in different lighting and seasons.

Figure 2:  A localization framework $f$ is view-invariant if and only if for
all transforms $(R,t)$ between the viewpoints $v(p)$ and $v^{\prime}(p)$ of
the point $p$ in the reference frame, it holds that
$f(v(p))=f(v^{\prime}(p))=p$.

Objects provide a common framework for localization using different sensing
modalities. For example, modern classifiers can detect the same object classes
in Lidar point clouds (e.g, PointNet [1], VoxelNet [2], RangeNet++ [3]) and in
images (e.g., YOLO [4], Mask R-CNN [5]). By registering the objects observed
in these modalities, a camera can be localized in a Lidar map. In this work,
each observed object is represented by a 3D point and its class. This compact
representation leads to a small map size for a large environment, and
consequently enables more efficient communication of the reference and vehicle
maps (this is crucial in a multi-agent setting with bandwidth constraints).

We demonstrate the framework experimentally for the case of localizing a
ground vehicle in a reference map, using only cars as semantic objects. Only
stereo camera images from the KITTI dataset [6] are used to create the ground
vehicle’s object map. We consider reference maps constructed from cars seen a
priori from ground viewpoints using stereo cameras and Lidar scans, and aerial
maps captured on different dates to demonstrate the framework’s robustness to
viewpoints, sensing modalities, and environment changes (see Fig. 1). The
localization performance is analyzed in terms of pose accuracy and convergence
time. With accuracy comparable to state-of-the-art methods, our framework is
also view-invariant and robust to environment changes.

In summary, the contributions of this work are:

  * •

View-invariant formulation of localization as the registration of semantic
objects, applicable to all cross-view scenarios with jointly observed objects
(see Fig. 2).

  * •

Resilient to environment/object changes due to a robust data association
framework capable of handling large outlier ratios in associations (>90%).

  * •

Demonstrations on real data from the KITTI dataset showing comparable accuracy
to state-of-the-art methods, cross-modal localization between Lidar and stereo
maps using objects, and geo-localization in an aerial map captured on a
different date with significantly changed objects.

While prior work may share a subset of above contributions, what distinguishes
this systems paper is their combination (as discussed in related work).

##  II Related Work

The large body of work on localization has close ties to loop closure
detection, place recognition, and image retrieval literature. We focus our
review on the most recent/related work in these domains.

###  II-A Appearance-Based Methods

Appearance-based methods use images for localization via finding the most
visually similar image in the reference database to the locally captured
image. The visual similarity is assessed based on low-level information such
as color and reflectance values [7, 8] or (low-level) visual
features/descriptors [9, 10]. Early works such as [11], [12] use local feature
descriptors to compare and match images taken from different perspectives.
Majdik et al. [13] use such features with simulated images from Google Street
View to match against images from a quadrotor flying through an urban
environment. Methods based on low-level features are impacted the most by
changes in the environment such as illumination or seasonal changes, and, more
importantly, fail under extreme viewpoint difference between the vehicle and
reference images (e.g., aerial-ground).

###  II-B Cross-View Localization

The term “cross-view” is mainly used by the literature for geo-localization of
a ground vehicle in an aerial/satellite map. However, some works also consider
localization of an aerial vehicle (e.g., a drone) in the satellite image [14].
Current learning-based state-of-the-art [15, 16, 14] use Siamese network
architecture [17, 18], and can return a coarse localization (accuracy of
hundreds of meters) across a very large area (e.g., city-wide). When coupled
with particle filters, these techniques can provide a higher accuracy (tens of
meters) in geo-tracking applications [19]. Due to the air-ground view, cross-
view algorithms are generally robust to viewpoint variations, however, they
are not necessarily view-invariant (see definition in Fig. 2). For example,
even though these methods work for the extreme air-ground view, they may not
work directly for other views (e.g., ground-ground). Further, many are
relatively slow for real-time robotic applications.

Air-ground localization can be achieved by other techniques, such as [20, 21,
22] that can obtain highly accurate localization (centimeter-level) when a
high-definition point cloud map of the environment is available. While
accurate, these methods do not work with image maps and require dense point
clouds.

###  II-C Semantic-Aided Methods

Several recent works have leveraged semantic information for localization.
Different types of semantic information used in the air-ground localization
literature include building outlines [23, 24], general vertical structures
[25, 26, 27], lane markings [28, 29], binary ground-nonground distinction
[30], vehicle trajectory [31], [32], semantic segmentation [10, 33], buildings
[34, 35, 36], intersections and building gaps [37], and 3D labeled points
[38], [39]. While semantic classifiers are typically robust to viewpoint
variations and to seasonal and lighting changes, using semantics limits the
applicability to environments where the desired classes are present (e.g.,
urban).

Figure 3: Main components of the tracking and localization framework. From
onboard sensors, the simultaneous localization and mapping (SLAM) system
estimates the vehicle’s pose in its local frame, the classifier identifies
objects, and the vehicle object map is built. The objects in the reference map
are grouped into submaps and the vehicle is localized by performing data
association to align the two maps.

###  II-D Placement of This Work

This work leverages semantic information, however, its view-invariant
formulation and ability to handle changes in the environment (in terms of
objects) distinguishes it from prior art. Methods that leverage semantic
information for localization have not shown to be robust to changes in the
semantic data nor completely view-invariant. Methods are typically designed
specifically for ground-ground, air-air, or air-ground localization, and the
ability of one framework to handle all cases is rarely analyzed. A subtler
difference is that instead of one-time localization, the framework is designed
for real-time continuous localization and tracking of the vehicle’s pose in
the environment, which is important for robotic applications. Only few works
[33, 40, 19] have considered this case.

##  III PIPELINE

Components of the proposed pipeline are shown in Fig. 3 and explained in
detail below.

###  III-A Onboard Sensors

Sensor measurements are used to estimate the vehicle’s motion and to detect
and reconstruct objects. Any sensor suite that can provide egomotion
estimation, object detection, and depth can be used (e.g., camera+IMU, Lidar);
in this work we experiment with a stereo camera.

###  III-B Reference Map and Submaps

The pipeline takes in a reference map of objects in the environment, which can
be fully constructed a priori or changing in real-time (e.g., dynamic
reference maps are important in multi-agent settings). Our experiments were
conducted on static maps. Objects can be of any type/class provided that a
small subset of objects detected by the vehicle are co-located with the
objects in the reference map, so they can be registered and thus provide
localization.

Each object is represented by its 3D centroid $p\in\mathbb{R}^{3}$ and an
integer type/class $c\in\mathbb{N}$. Therefore, the reference map can be
denoted as a set of tuples
$S_{\text{ref}}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\{(p_{i},c_{i}):\,p_{i}\in\mathbb{R}^{3},\,c_{i}\in\mathbb{N},i=1,\dots,n\\}$.
Since the computational complexity of the registration module grows with the
number of objects (Sec. III-F), to achieve real-time registration, the
reference map can be divided into $k$ smaller submaps
$S^{i}_{\text{ref}}\subset S_{\text{ref}}$, $i=1,\dots,k$. The submaps may
overlap and they must cover the entire reference map (i.e.,
$\bigcup_{i}S^{i}_{\text{ref}}=S_{\text{ref}}$).

###  III-C SLAM System

The purpose of the SLAM module is to provide odometry estimates for the
vehicle and to reconstruct a point cloud map of the environment. Our
implementation uses ORB-SLAM3 [41], which takes in the stereo camera images as
input and returns the vehicle’s pose and 3D reconstructed ORB feature points.
This output is used subsequently to reconstruct the objects and track the
egomotion of the vehicle in the reference map.

###  III-D Classifier

The role of the classifier is to detect semantic objects of interest in the
sensor data. We only consider cars as objects in our demonstrations, however,
other types of objects can be used instead (e.g., traffic signs, trees).
Provided an RGB image as input, we use YOLO-V3 [42] to return a bounding box
and class for each detected object. This bounding box is then used to estimate
the object’s centroid.

Figure 4:  (Left to right) KITTI Sequences 0, 2, and 9. The localized
trajectory of the ground vehicle (red line), vehicle map (red squares), and
reference map (blue squares) constructed from the background orthophotos
captured on a different date are shown. The background images are provided for
reference and only the cars (blue and red squares) are used for registration.

###  III-E Object Mapping

An object’s centroid is estimated using the 3D landmarks reconstructed by the
SLAM system that correspond to the object, meaning the 3D reconstructed
landmarks that project inside the object’s bounding box in the image. We
approximate the distance to the object from the current vehicle pose by
averaging the distance to all corresponding ORB landmarks. Given this
distance, we represent the object centroid with the 3D point that is at the
computed distance and projects onto the center of the bounding box.
Subsequently, the vehicle object map is constructed by tracking all object
centroids across consecutive camera frames and fusing them into one map. We
denote the vehicle object map by the set of 3D points and their classes as
${S_{\text{loc}}\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\{(q_{i},c_{i}):\,q_{i}\in\mathbb{R}^{3},\,c_{i}\in\mathbb{N},i=1,\dots,m\\}}$.
Instead of using the entire vehicle map, a subset of the recently observed
objects is used for localization. This speeds up the registration by reducing
the problem size and reduces the registration errors that are due to the
accumulated drift in the SLAM trajectory.

###  III-F Registration

Robust registration is the core component of the proposed framework. We use a
graph-based formulation to solve the registration problem by finding the
largest set of geometrically consistent objects that match between the
reference submaps and vehicle map. Denoting by $a_{i}=(p_{i},q_{i})$ the
association that matches the points $p_{i}$ and $q_{i}$, two associations
$a_{i}$ and $a_{j}$ are considered geometrically consistent if and only if the
distance between the points is preserved, i.e.,
$\|p_{i}-p_{j}\|=\|q_{i}-q_{j}\|$. In practice, due to noise and inaccuracies,
a threshold $\epsilon$ is set in order to consider associations consistent
when $d(a_{i},a_{j})\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}|\,\|p_{i}-p_{j}\|-\|q_{i}-q_{j}\|\,|<\epsilon$. Now, by denoting
the set of associations between objects of the same class in the reference and
vehicle maps as ${A\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\\{(o_{i},o_{j}):\,o_{i}\in S_{\text{ref}},\,o_{j}\in
S_{\text{loc}},\,\text{class}(o_{i})=\text{class}(o_{j})\\}}$, the problem of
finding the largest set of consistent associations, $A^{*}_{\text{c}}$, can be
defined formally as

$\displaystyle\begin{array}[]{ll}\underset{A_{\text{c}}\subset
A}{\text{maximize}}&|A_{\text{c}}|\\\ \text{subject
to}&d(a_{i},a_{j})<\epsilon,~{}\forall_{a_{i},a_{j}\in
A_{\text{c}}}.\end{array}$ (3)

Figure 5: Maximum clique formulation for registration. (Left) Points $p$ and
$q$ are matched by associations $a$. (Right) Graph with nodes representing
associations and edges indicating their consistency (i.e., associations with
identical distances between their endpoints). The largest clique
${A^{*}_{\text{c}}=\\{a_{1},a_{2},a_{4}\\}}$ is the largest set of consistent
associations.

Problem (3) can be modeled as a graph with associations representing the graph
nodes and edges representing consistent associations. The optimal solution of
(3) is equivalent to the maximum clique of the graph. Fig. 5 illustrates this
point. Finding the maximum clique is, in general, an NP-hard problem, which
implies that exact algorithms become computationally inhibiting as the graph
size grows. However, for sparse graphs (resulting from many inconsistent
associations created by an all-to-all scheme), the problem can be solved
relatively quickly using the parallel maximum clique (PMC) algorithm [43].

We use an all-to-all association scheme, i.e., we initially associate an
object in the reference map to every object in the vehicle map (of the same
class) and vice versa. Despite the large number of association outliers
generated by the all-to-all association and by inconsistent points, solving
(3) using PMC is guaranteed to return the maximum set of consistent
associations. The all-to-all scheme ensures that geometrically consistent
objects are matched from any configuration, hence the view-invariant claim.
The least-square fitting of matched objects using Arun’s method [44] gives the
optimal transformation $(R,t)\in\text{SO}(3)$ that registers the objects.

###  III-G Localization

The final module in the pipeline is localization. This module continuously
tracks the vehicle’s pose in the reference map by transforming the vehicle’s
pose in the SLAM frame to the reference frame using the latest registration
results. The highest number of matched points in all previous registrations is
kept and a new registration is only accepted when the number of matched points
is 90% or higher than the highest quality registration from the history.

##  IV EXPERIMENTAL EVALUATIONS

We demonstrate the framework experimentally by localizing a ground vehicle in
a reference map. We use stereo camera images from the KITTI dataset [6] to
create the ground vehicle’s object map, as this dataset is used by several
prior works and provides a benchmark for comparison. The framework is
implemented in C++ and modules use the robot operating system (ROS) [45] to
communicate. The pipeline runs real-time on a laptop with an i7-8750H CPU, 16
GB RAM, and Nvidia 1060 GPU. The objectives are to evaluate our framework with
different map modalities, different viewpoints, environment/object changes,
and to perform an ablation study of errors.

###  IV-A Evaluation Scenarios

To create a particularly challenging environment, we only consider cars as
semantic objects (as opposed to buildings [33] or lane markings [28, 29]
considered in other work). This enables us to test the framework with a
significantly changed semantic object map because cars are dynamic or quasi-
static (i.e., parked cars are at different locations on different dates). It
is possible to localize a vehicle in a reference map which contains none of
the same cars as in the vehicle map because cars are a proxy for parking
spaces and our framework is indifferent to visual car attributes (e.g., color,
model) for registration. We consider KITTI sequences numbered 0, 2, and 9
since they have been used by prior art as benchmarks [33]. Algorithm
parameters are tuned on Sequence 0, and the same values are used for other
sequences. In particular, we use a threshold of $\epsilon=5$ m for
registration (defined in Sec. III-F) and restrict the size of the vehicle
object map to the last $75$ seen objects/cars to mitigate the effect of drift.
We consider registration results that match at least $20$ objects across the
reference and vehicle maps, but this number may increase as more accurate
registrations are found. Due to the larger size of the reference map for KITTI
Sequence 0, it is grouped into $4$ submaps with $50\%$ overlap, but no submaps
are used for KITTI Sequences 2 and 9.

For the reference map, we consider three scenarios of object/car maps created
from 1) ground-view stereo camera images, 2) ground-view Lidar scans, and 3)
aerial-view georeferenced images (orthophotos) to test the framework’s
robustness to viewpoints, sensing modalities, and environment changes that
exist across these maps. Fig. 6 shows an overlay of these object maps on the
area corresponding to Sequence 0, where cars are represented by their
centroids as 3D points. The aerial reference map presents the most challenging
case because it has the largest number of objects, the highest object outlier
ratio, is constructed from a different viewpoint than the vehicle, and the
orhtophotos were captured on a different date. Fig. 4 illustrates snippets of
the localization results in the aerial-ground case.

Figure 6: Reference object maps corresponding to KITTI Sequence 0 constructed
using stereo camera images (ground view), Lidar scans (ground view), and
Bing’s georeferenced images (aerial view). Each square represents a car. The
bottom image is for reference.

###  IV-B Baseline

To create a baseline case which factors out the effects of viewpoint, sensing
modality, and environment changes, we localize the ground vehicle in a
reference map built from the identical data (KITTI stereo images) as used by
the ground vehicle. This reference map (shown in Fig. 6) is created as
described in Sec. III-E, but using the ground truth vehicle pose instead of
the ORB-SLAM estimated pose. While the reference map consists of $374$ cars,
the vehicle’s map has $437$ cars because the object mapping module
misidentifies previously-seen objects as new instances due to drift in SLAM
pose estimates. The error in the pose estimates also affects the object
centroid estimation. For these reasons, despite identical sensing modalities
and viewpoints, only $179$ of the $437$ cars seen by the vehicle match to a
car in the reference map (based on the $\epsilon=5$ consistency threshold).
Thus, the vehicle map contains $59\%$ object outliers.

The localization time is defined as the time it takes from the start of the
sequence until the vehicle finds a correct registration, which is $32$ sec for
this case. The position and orientation errors are computed at each time step
after the localization time via
${e_{p}(t)\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny def}}}{=}}\parallel
T_{\text{gt}}(t)-T(t)\parallel}$ and
${e_{o}(t)\mathrel{\overset{\makebox[0.0pt]{\mbox{\tiny
def}}}{=}}\frac{180}{\pi}\arccos{(\mid<Q_{\text{gt}}(t),Q(t)>\mid)}}$, where
$T$ and $Q$ represent position and quaternion orientation, and “gt” denotes
the KITTI ground truth. Fig. 7 shows the error statistics. The pipeline
successfully rejects outlier objects and obtains an average localization error
of $6.1$ m.

Figure 7: Error statistics for localization of the ground vehicle in KITTI
Sequence 0 on three different reference maps (shown in Fig. 6). Errors are
comparable across cases, showing that our framework is not affected by
different object outlier percentages, map modalities, or viewpoints. The
orientation errors have large tails due to temporary error spikes in ORB-SLAM
orientation estimates during an acute turn in the trajectory.

###  IV-C Robustness to Map Modality

To analyze the effect of map modality, an object map is created from the KITTI
ground truth poses and the Lidar scans of a Velodyne Lidar mounted on the
ground vehicle (shown in Fig. 6). Here, each car centroid in the reference map
is computed as the median of the Lidar points associated with that car, as
labeled in the SemanticKITTI dataset [46]. The vehicle map has an object
outlier ratio of $56\%$ with respect to the Lidar map, which consists of $445$
cars. Since both the stereo baseline and Lidar reference maps have similar
object outlier ratios ($59\%$ and $56\%$) and are constructed from the same
ground truth poses, comparison of the localization errors isolates the effect
of sensing modality. The error statistics are shown in Fig. 7. The average
position errors are $6.1$ m and $8.5$ m and the localization times are $32$
sec and $36$ sec for the stereo-stereo and stereo-Lidar cases, respectively.
The errors are comparable, which indicates robustness of the pipeline to
sensing modalities. The small difference in errors is due to the different
centroid estimates for the same car in the Lidar and baseline reference maps
as centroids are computed using different methods.

TABLE I:  Statistics of KITTI ground vehicle localization in Bing’s georeferenced aerial images using only cars as semantic objects. Since the aerial reference map is 2D, the position and orientation errors are calculated after the pose estimate is projected onto the reference map plane.  Reference-Vehicle Viewpoint |  KITTI Sequence $[$#$]$ |  |  Average Position Error $[$m$]$ |  |  Average Orientation Error $[$deg$]$ |  |  Time to Localize $[$sec$]$ |  |  Distance to Localize $[$m$]$ |  |  Trajectory length $[$m$]$ |  |  Objects to Localize $[$#$]$ |  |  Objects in Reference Map $[$#$]$ |  |  Objects Seen by Vehicle $[$#$]$ |  |  Object Outliers $[$%$]$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Aerial-Ground |  $0$ $2$ $9$ |  |  $7.9$ $10.6$ $10.9$ |  |  $1.93$ $1.03$ $1.08$ |  |  $71$ $183$ $108$ |  |  $476$ $1836$ $1077$ |  |  $3724$ $5067$ $1705$ |  |  $45$ $81$ $56$ |  |  $675$ $409$ $391$ |  |  $418$ $200$ $97$ |  |  $77$ $85$ $77$

###  IV-D Robustness to Viewpoint and Environment Changes

To demonstrate that the framework is view-invariant, we consider air-ground
localization using a reference map created from Bing aerial images [47]. This
map has a fixed scale since orthophotos are georeferenced with the EPSG:3857
coordinates, which provide an accuracy of (at best) 2 meters. We annotated the
cars manually in aerial images (which can be automated by classifiers trained
for aerial/satellite images [48, 49]). The vehicle map has $77\%$ object
outliers with respect to the aerial map, consisting of $675$ cars (shown in
Fig. 6). The localization error statistics are reported in Fig. 7.

Compared to the stereo and Lidar maps, the aerial reference map provides the
most challenging localization scenario because the orthophotos and the KITTI
dataset are captured in different years and the cars are unlikely to be
identical and/or at the same location. Further, some cars could not be labeled
in the aerial images due to natural occlusions (e.g., trees, shadows) and low
image resolution. Despite the limited $2$ m accuracy of the EPSG coordinates
and challenges above, the pipeline successfully localized the ground vechicle
in $71$ sec and with an average position error of $7.9$ m. This error is
comparable to the baseline and stereo-Lidar cases ($6.1$ m and $8.5$ m), which
demonstrates that the framework is invariant to viewpoints and changes in
environment objects.

Figure 8: Air-ground localization error in KITTI Sequence 0 by using stereo
pose estimates (green) and ground truth pose (yellow). The latter shows the
highest attainable localization accuracy, e.g., using a highly-accurate SLAM
solution. The occasional lower error from stereo corresponds to instances
where the vehicle pose error counteracts the registration error.

###  IV-E Maximum Localization Accuracy

The accuracy of vehicle pose estimates depends on various factors such as
sensors (e.g., camera and Lidar measurements have different noise models that
lead to different accuracy), number of loop closures, and robustness of the
backend optimizer in the SLAM module. Pose estimates affect object mapping and
egomotion tracking, and therefore directly influence the localization accuracy
in our pipeline. The localization accuracy is also affected by registration.

Fig. 8 analyzes these errors by replacing the ORB-SLAM pose estimates with the
the ground truth vehicle pose for localization in the aerial map. The
localization error using the ground truth is only due to registration. This
case has an average position error of $4.1$ m compared to the $7.9$ m from
stereo pose estimates. This scenario shows the highest localization accuracy
that is achievable by the pipeline, which, for example, can be approached by
using a highly-accurate SLAM solution (e.g., top-ranked solutions in the KITTI
odometry benchmark [50]). The registration error is due to inaccuracies in
object centroid estimates in both the ground vehicle map and the aerial
reference aerial map (our aerial map has an accuracy of at best $2$ m).

###  IV-F Comparison with Prior Art

Table I lists the evaluation results for the real-time localization of a car
with an onboard stereo camera (KITTI dataset) in georeferenced aerial images
shown in Fig. 4. Table II compares these results with prior art that similarly
used the KITTI benchmark for air-ground localization. The results are carried
over here as reported in the corresponding papers since open-source code was
not available. We report the localization error for both cases of stereo
odometry and the ground truth odometry, which provides the maximum achievable
accuracy of our pipeline.

While the prior state-of-the-art achieves remarkable accuracy, it is unclear
if/how these techniques perform in the highly changed environments since the
semantic information used (e.g., buildings, road maps, traffic signs) did not
change considerably across views in their evaluations. In contrast, the
challenging scenario created by using cars in an aerial map captured on a
different date demonstrates the robustness of our approach in a very different
environment leading to comparable localization accuracy in real time.

TABLE II:  Aerial-ground localization comparison on KITTI benchmark for
position error and localization time. Dashed line “$-$” indicates not reported
or not localized successfully.

Approach |  Ground Modality |  |  Aerial Modality |  | Seq 0 |  | Seq 2 |  | Seq 9
---|---|---|---|---|---|---|---|---|---
|  |  Error $[$m$]$ |  Time $[$sec$]$ |  |  Error $[$m$]$ |  Time $[$sec$]$ |  |  Error $[$m$]$ |  Time $[$sec$]$
Fervers [40] | L+C |  | OP |  | $-$ | $-$ |  | $1.42$ | $-$ |  | $-$ | $-$
Miller [33] | L+C |  | OP |  | $2.0$ | $54.6$ |  | $9.1$ | $71.5$ |  | $7.2$ | $75$
Yan [37] | L |  | OSM |  | $20$ | $60$ |  | $-$ | $-$ |  | $25$ | $17$
Kim [24] | L |  | OP |  | $4.6$ | $62$ |  | $-$ | $-$ |  | $7.7$ | $55.6$
Brubaker [31] | S |  | OSM |  | $2.1$ | $-$ |  | $4.1$ | $-$ |  | $4.2$ | $-$
Floros [32] | C |  | OSM |  | $>10$ | $-$ |  | $>20$ | $-$ |  | $-$ | $-$
Ours | GT |  | OP |  | $3.9$ | $71$ |  | $9.1$ | $136$ |  | $7.4$ | $86$
Ours | S |  | OP |  | $7.9$ | $71$ |  | $10.6$ | $183$ |  | $10.9$ | $108$

C: Monocular Camera, S: Stereo Camera, L: Lidar, GT: Ground Truth
OP: Orthophoto, OSM: OpenStreetMap

##  V DISCUSSION

The following discusses some practical limitations and possible extensions of
the framework. 1) The localization framework is based on registering
geometrically consistent objects, which must be detectable in the vehicle and
reference viewpoints. Thus, while in theory the localization framework is
view-invariant, in practice, viewpoint variations can be constrained by the
limitations of object classifiers. 2) Environments with repetitive/symmetrical
object patterns can create scenarios with a large number of geometrically
consistent false matches (e.g., a long line of cars can register to another
line on a different street). While repetitive patterns (perceptual aliasing)
are a fundamental issue in place recognition/localization, steps can be taken
to alleviate the problem. For example, object types which have a more random
pattern in the environment can be used. 3) Geometric consistency implies that
the object maps must have a well-defined scale, which is not the case in
monocular SLAM. It is possible to extend the formulation using scale-free
metrics (such as geometric similarity) at the cost of higher computational
complexity. 4) We used a point cloud representation of object centroids, which
had sufficient accuracy for car-based localization. However, this
representation may not be sufficient for larger objects such as buildings.
Hence, other geometric representations such as lines or planar patches should
be used instead, and the formulation of geometric consistency can be extended
to these representations using proper invariants (cf. prior works [51, 52]).

##  VI CONCLUSIONS

We presented a view-invariant framework for real-time localization and
continuous tracking of a vehicle’s pose using compact semantic object maps.
Experiments localizing a ground vehicle in reference maps created by stereo
cameras and Lidar scans from ground viewpoints, and annotated georeferenced
aerial images captured in a different year demonstrated the main contributions
of the framework as a view-invariant approach robust to environment object
changes.

## References

  * [1] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 652–660.
  * [2] Y. Zhou and O. Tuzel, “VoxelNet: End-to-end learning for point cloud based 3D object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4490–4499.
  * [3] A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, “RangeNet++: Fast and accurate lidar semantic segmentation,” in _2019 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 2019, pp. 4213–4220.
  * [4] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [5] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in _Proceedings of the IEEE International Conference on Computer Vision_ , Oct 2017.
  * [6] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite,” in _Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition_ , 2012, pp. 3354–3361.
  * [7] L. D. P. Veronese, E. de Aguiar, R. C. Nascimento, J. Guivant, F. A. A. Cheein, A. F. De Souza, and T. Oliveira-Santos, “Re-emission and satellite aerial maps applied to vehicle localization on urban environments,” in _2015 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 2015, pp. 4285–4290.
  * [8] A. Vora, S. Agarwal, G. Pandey, and J. McBride, “Aerial imagery based lidar localization for autonomous vehicles,” _arXiv preprint arXiv:2003.11192_ , 2020.
  * [9] T. Senlet and A. Elgammal, “A framework for global vehicle localization using stereo images and satellite and road maps,” in _2011 IEEE International Conference on Computer Vision Workshops_. IEEE, 2011, pp. 2034–2041.
  * [10] M. Noda, T. Takahashi, D. Deguchi, I. Ide, H. Murase, Y. Kojima, and T. Naito, “Vehicle ego-localization by matching in-vehicle camera images to an aerial image,” in _Asian Conference on Computer Vision_. Springer, 2010, pp. 163–173.
  * [11] D. M. Chen, G. Baatz, K. Köser, S. S. Tsai, R. Vedantham, T. Pylvänäinen, K. Roimela, X. Chen, J. Bach, M. Pollefeys _et al._ , “City-scale landmark identification on mobile devices,” in _Conference on Computer Vision and Pattern Recognition_. IEEE, 2011, pp. 737–744.
  * [12] Y. Li, N. Snavely, and D. P. Huttenlocher, “Location recognition using prioritized feature matching,” in _European conference on computer vision_. Springer, 2010, pp. 791–804.
  * [13] A. L. Majdik, Y. Albers-Schoenberg, and D. Scaramuzza, “Mav urban localization from google street view data,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 3979–3986.
  * [14] T. Wang, Z. Zheng, C. Yan, J. Zhang, Y. Sun, B. Zheng, and Y. Yang, “Each part matters: Local patterns facilitate cross-view geo-localization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 2, pp. 867–879, 2021.
  * [15] Y. Shi, L. Liu, X. Yu, and H. Li, “Spatial-aware feature aggregation for image based cross-view geo-localization,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [16] R. Rodrigues and M. Tani, “Are these from the same place? seeing the unseen in cross-view image geo-localization,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 3753–3761.
  * [17] D.-K. Kim and M. R. Walter, “Satellite image-based localization via learned embeddings,” in _2017 IEEE International Conference on Robotics and Automation_ , 2017, pp. 2073–2080.
  * [18] Y. Tian, X. Deng, Y. Zhu, and S. Newsam, “Cross-time and orientation-invariant overhead image geolocalization using deep local features,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 2512–2520.
  * [19] L. M. Downes, D.-K. Kim, T. J. Steiner, and J. P. How, “City-wide street-to-satellite image geolocalization of a mobile ground agent,” _arXiv preprint arXiv:2203.05612_ , 2022.
  * [20] R. W. Wolcott and R. M. Eustice, “Visual localization within lidar maps for automated urban driving,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 176–183.
  * [21] A. Gawel, R. Dubé, H. Surmann, J. Nieto, R. Siegwart, and C. Cadena, “3D registration of aerial and ground robots for disaster response: An evaluation of features, descriptors, and transformation estimation,” in _2017 IEEE International Symposium on Safety, Security and Rescue Robotics_ , 2017, pp. 27–34.
  * [22] I. A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun, “Learning to localize using a lidar intensity map,” _arXiv preprint arXiv:2012.10902_ , 2020.
  * [23] O. Vysotska and C. Stachniss, “Improving slam by exploiting building information from publicly available maps and localization priors,” _PFG–Journal of Photogrammetry, Remote Sensing and Geoinformation Science_ , vol. 85, no. 1, pp. 53–65, 2017.
  * [24] J. Kim and J. Kim, “Fusing lidar data and aerial imagery with perspective correction for precise localization in urban canyons,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2019, pp. 5298–5303.
  * [25] R. Kümmerle, B. Steder, C. Dornhege, A. Kleiner, G. Grisetti, and W. Burgard, “Large scale graph-based slam using aerial images as prior information,” _Autonomous Robots_ , vol. 30, no. 1, pp. 25–39, 2011.
  * [26] X. Wang, S. Vozar, and E. Olson, “Flag: Feature-based localization between air and ground,” in _2017 IEEE international conference on robotics and automation_. IEEE, 2017, pp. 3178–3184.
  * [27] H. Wang, C. Xue, Y. Tang, W. Li, F. Wen, and H. Zhang, “LTSR: Long-term Semantic Relocalization based on HD Map for Autonomous Vehicles,” in _2022 International Conference on Robotics and Automation_. IEEE, 2022, pp. 2171–2178.
  * [28] O. Pink, “Visual map matching and localization using a global feature map,” in _2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops_. IEEE, 2008, pp. 1–7.
  * [29] M. Javanmardi, E. Javanmardi, Y. Gu, and S. Kamijo, “Towards high-definition 3d urban mapping: Road feature-based registration of mobile mapping systems and aerial imagery,” _Remote Sensing_ , vol. 9, no. 10, p. 975, 2017.
  * [30] A. Viswanathan, B. R. Pires, and D. Huber, “Vision-based robot localization across seasons and in remote locations,” in _2016 IEEE international conference on robotics and automation_. IEEE, 2016, pp. 4815–4821.
  * [31] M. A. Brubaker, A. Geiger, and R. Urtasun, “Lost! leveraging the crowd for probabilistic visual self-localization,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2013, pp. 3057–3064.
  * [32] G. Floros, B. Van Der Zander, and B. Leibe, “Openstreetslam: Global vehicle localization using openstreetmaps,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 1054–1059.
  * [33] I. D. Miller, A. Cowley, R. Konkimalla, S. S. Shivakumar, T. Nguyen, T. Smith, C. J. Taylor, and V. Kumar, “Any way you look at it: Semantic crossview localization and mapping with lidar,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2397–2404, 2021.
  * [34] B. C. Matei, N. Vander Valk, Z. Zhu, H. Cheng, and H. S. Sawhney, “Image to lidar matching for geotagging in urban environments,” in _2013 IEEE Workshop on Applications of Computer Vision_. IEEE, 2013, pp. 413–420.
  * [35] T. Senlet, T. El-Gaaly, and A. Elgammal, “Hierarchical semantic hashing: Visual localization from buildings on maps,” in _2014 22nd International Conference on Pattern Recognition_. IEEE, 2014, pp. 2990–2995.
  * [36] Y. Tian, C. Chen, and M. Shah, “Cross-view image matching for geo-localization in urban environments,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3608–3616.
  * [37] F. Yan, O. Vysotska, and C. Stachniss, “Global localization on OpenStreetMap using 4-bit semantic descriptors,” in _2019 European Conference on Mobile Robots_. IEEE, 2019, pp. 1–7.
  * [38] E. Stenborg, C. Toft, and L. Hammarstrand, “Long-term visual localization using semantically segmented images,” in _2018 IEEE international conference on robotics and automation_. IEEE, 2018, pp. 6484–6490.
  * [39] Y. Liu, Y. Petillot, D. Lane, and S. Wang, “Global localization with object-level semantics and topology,” in _2019 International Conference on Robotics and Automation_. IEEE, 2019, pp. 4909–4915.
  * [40] F. Fervers, S. Bullinger, C. Bodensteiner, M. Arens, and R. Stiefelhagen, “Continuous self-localization on aerial images using visual and lidar sensors,” _arXiv preprint arXiv:2203.03334_ , 2022.
  * [41] C. Campos, R. Elvira, J. J. G. Rodrίguez, J. M. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [42] M. Bjelonic, “YOLO ROS: Real-time object detection for ROS,” https://github.com/leggedrobotics/darknet_ros, 2016–2018.
  * [43] R. A. Rossi, D. F. Gleich, A. H. Gebremedhin, M. Patwary, and M. Ali, “Parallel maximum clique algorithms with applications to network analysis and storage,” _arXiv preprint arXiv:1302.6256_ , 2013.
  * [44] K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-d point sets,” _IEEE Transactions on pattern analysis and machine intelligence_ , no. 5, pp. 698–700, 1987.
  * [45] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng _et al._ , “ROS: an open-source Robot Operating System,” in _ICRA workshop on open source software_ , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5.
  * [46] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, “SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences,” in _Proc. of the IEEE/CVF International Conf. on Computer Vision_ , 2019.
  * [47] Bing aerial images, https://www.bing.com/maps/aerial.
  * [48] J. Ding, N. Xue, Y. Long, G. Xia, and Q. Lu, “Learning roi transformer for detecting oriented objects in aerial images,” in _The IEEE Conference on Computer Vision and Pattern Recognition_ , June 2019.
  * [49] W. Li, Y. Chen, K. Hu, and J. Zhu, “Oriented reppoints for aerial object detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 1829–1838.
  * [50] KITTI odometry benchmark, https://www.cvlibs.net/datasets/kitti/eval_odometry.php.
  * [51] P. C. Lusk, K. Fathian, and J. P. How, “CLIPPER: A Graph-Theoretic Framework for Robust Data Association,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 13 828–13 834.
  * [52] P. C. Lusk and J. P. How, “Global Data Association for SLAM with 3D Grassmannian Manifold Objects,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022.