# RenderNet: Visual Relocalization Using Virtual Viewpoints in Large-Scale
Indoor Environments

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Visual relocalization has been a widely discussed problem in 3D vision: given
a pre-constructed 3D visual map, the 6 DoF (Degrees-of-Freedom) pose of a
query image is estimated. Relocalization in large-scale indoor environments
enables attractive applications such as augmented reality and robot
navigation. However, appearance changes fast in such environments when the
camera moving, which is challenging for the relocalization system. To address
this problem, we propose a virtual view synthesis based approach, RenderNet,
to enrich the database and refine poses regarding this particular scenario.
Instead of rendering real images which requiring high-quality 3D models, we
opt to directly render the needed global and local features of virtual
viewpoints and apply them in the subsequent image retrieval and feature
matching operations respectively. The proposed method outperforms state-of-
the-art visual relocalization methods by a large margin on two large-scale
indoor public datasets, InLoc and RIO, only using the existing database
information.

##  1 Introduction

In order to satisfy the growing demand for various spatial intelligence
relevant applications, such as navigation, Augmented Reality (AR) and SLAM,
visual relocalization is an indispensable technology. It is essentially a
registration problem to localize the query images within a visual database. By
establishing 2D-3D correspondences between query images and database images, 6
DoF camera poses of the query images can be estimated.

Current state-of-the-art scene agnostic localization methods
[sattler2018benchmarking, sarlin2019coarse] are comprised of an image
retrieval module searching for candidate database images as the localization
reference, and an image matching module to establish 2D-3D correspondences.
Then, the camera poses are computed through a subsequent RANSAC + Perspective-
n-Point (PnP) algorithm. However, the performance of both modules suffers from
large view and appearance changes, which are common in large-scale indoor
environments, since appearance changes rapidly when the camera moves. As shown
in Fig. 1, the most similar images in the database might still have
insufficient overlap with the query image, which are hard to be retrieved and
matched, thus resulting in degraded performance.

Figure 1: Comparison of the state-of-the-art approach HLoc [sarlin2019coarse]
and the proposed RenderNet. The top image pair shows retrieved real database
image while the bottom image pair shows virtual synthesis views. Putative
matches from SuperGlue matcher [sarlin2020superglue] are shown. The virtual
view, which is not explicitly generated, can have a larger overlap with the
query image than real database images, thus can be helpful for both image
retrieval and image matching and result in better relocalization performance.

In order to address this problem, an intuitive idea is to synthesis novel
views to enrich the database. Database images with larger overlaps make image
retrieval and image matching easier. Thus the overall relocalization
performance can be improved accordingly. However, image synthesis requires
high-quality 3D models, which are usually captured by high-cost 3D scanners
and this process is time-consuming. For example, the Inloc
[wijmans2017exploiting] relocalization dataset is captured using an expensive
Lidar scanner [faro] which requires more than 30 minutes for a single scan.

Instead of rendering novel views from a pre-built high-quality 3D model, we
propose RenderNet to generate virtual novel views from the existing database
images. Different from the general view synthesis process [martin2020nerf,
nakata2002synthesis] that aims to render visual-realistic images, the proposed
neural network learns to directly obtain global features and local features of
virtual viewpoints used in the image retrieval module and image matching
module respectively. We opt to generate the needed features, which are the
final quantities for localization, instead of generating images, to reduce the
influence of inevitable artifacts in the synthesized images.

With the generated virtual views, we optimize the traditional localization
pipeline by plugging in two operations, namely the view augmentation operation
and pose refinement operation. In the view augmentation stage, we manually
pre-sample a set of possible views in a scene as virtual views, so that query
images can retrieve those views with their rendered global features, and
correspondences can be established via local features. In the pose refinement
stage, a new virtual view is generated using the coarse camera pose computed
in the previous stage. Then 2D-3D correspondences can be matched and refined
poses are obtained subsequently.

Those generated virtual views can largely improve the image retrieval and
correspondences establishing process. Our method is suitable for indoor scenes
where images change rapidly when viewpoints change. In the experimental
section, we will demonstrate that the proposed approach can largely improve
the baseline localization method, (7.1% and 12.2% of accuracy improvement in
the two scenes of Inloc).

##  2 Related work

#### Visual relocalization.

Visual relocalization aims to predict 6 DoF camera poses in a pre-constructed
3D map from RGB images. Current localization pipelines can be divided into
three categories: 1) End-to-end pose regression methods [kendall2015posenet,
kendall2017geometric, brahmbhatt2018geometry, kendall2016modelling], which
directly regress absolute camera pose from a single RGB image. These methods
are later shown to be closely related to image retrieval
[sattler2019understanding]. 2) Coordinate regression [schohn2000less,
brachmann2017dsac, brachmann2019expert] methods regress the dense 3D scene
coordinates of the query image and obtain the final camera pose by dense 2D-3D
correspondences. Most of these methods are scene-specific and need to be
trained for a new scene. 3) Image matching based-methods. These methods first
retrieve similar images from the database by image retrieval [gordo2016deep,
arandjelovic2016netvlad], then establish sparse [sarlin2019coarse] or dense
[rocco2018neighbourhood] correspondences between query images and retrieved
database images to form the 2D-3D correspondences. Poses are finally solved by
PnP algorithms. Current state-of-the-art methods for large-scale localization
are mostly based on image matching. There are many works focus on local
feature learning [detone2018superpoint, dusmanu2019d2, luo2018geodesc,
luo2019contextdesc, luo2020aslfeat], sparse feature matching [yi2018learning,
zhang2019learning, sarlin2020superglue] and dense matching
[rocco2018neighbourhood, rocco2020efficient] to improve the performance of
image matching based localization methods.

However, image matching-based methods still suffer from significant
performance drop if the database images have small overlaps with query images,
which is common in large-scale environments. Our RenderNet eases this problem
by generating additional virtual views.

#### View synthesis and pose estimation.

View synthesis aims to generate photo-realistic images of arbitrary novel view
given 3D models, sparse views [martin2020nerf] or point cloud
[sibbing2013sift, pittaluga2019revealing]. View synthesis has been shown
useful in many pose estimation related-areas. Torii et al. [torii201524] show
that image retrieval performance can be enhanced by enriching the database
with novel views. Sibbing et al. [sibbing2013sift] render view from point
clouds for visual localization and Zhang et al. [zhang2020reference] refine
the estimated poses of localization by view synthesis.

Our proposed RenderNet is inspired by the above applications of view
synthesis. However, instead of direct rendering photo-realistic images,
RenderNet generates virtual views by directly generating the needed features
used in the localization system, which will be shown can be more robust than
the direct view synthesis approach.

##  3 Visual relocalization by virtual viewpoints

Figure 2: The overall pipeline of the proposed visual relocalization approach.
Top left: virtual views are generate by projected colors and local features
from nearby real images. Top right: virutal views are used together with real
database images for view augmentation. To be specific, RetrievalNet and
DescNet are used for both query images and real database images, to generate
global feature $G_{q},G_{r}$ and local features $L_{q},L_{r}$ for image
retrieval and image matching respectively. $\text{RenderNet}_{\text{G}}$ and
$\text{RenderNet}_{\text{L}}$ are used for the virtual view to generate
corresponding global feature $G_{v}$ and local features $L_{v}$. Bottom: pose
refinement to improve relocalization performance. The query pose is first
estimated using RANSAC + PnP and further refined by taking the virtual view at
the initial pose as the localization reference.

We first give an introduction to our relocalization method which uses virtual
viewpoints for view augmentation and pose refinement. Then we present the
details of RenderNet, which generates global and local features of virtual
views used in the image retrieval and feature matching stage of the
relocalization pipeline.

###  3.1 Overview

Our method follows the traditional paradigm which first retrieves similar
images from database images by measuring the distance of their global
features, and then estimates camera poses through 2D-3D correspondences from
local feature matching. Besides, we generate virtual views to improve the
image retrieval and image matching process. With the generated novel view
features, we design view augmentation and pose refinement stages to improve
localization performance, as shown in Fig 2.

View augmentation stage. In this stage, we enrich the database with rendered
virtual views on a set of manually defined poses, which could have larger
overlaps with queries than existing real database images. We first define a
set of novel views around existing viewpoints and then generate their
corresponding global and local features. As shown in Fig. 2, for query images
and real database images, we generate global feature vectors using off-the-
shelf retrieval networks (RetrievalNet in Fig. 2), such as AP-GeM
[revaud2019learning]. In addition, interest points are detected and described
using local feature networks (DesNet in Fig. 2), such as SuperPoint
[detone2018superpoint]. For virtual views, we generate their global feature
vectors using our $\text{RenderNet}_{\text{G}}$ and their local features using
$\text{RenderNet}_{\text{L}}$. In the retrieval stage, query images find the
most similar top k database images by measuring the distance of global
features. Real images and virtual images are treated equally here. After that,
we can establish 2D-3D correspondences between query images and retrieved
images through sparse feature matching. Finally, RANSAC + PnP is performed to
estimate a coarse camera pose from established correspondences.

Pose refinement stage. To refine the estimated camera pose, we render a new
virtual view at the coarse pose estimated by the above localization pipeline,
and then perform sparse matching and PnP pose estimation again. We generate
only local features this time using $\text{RenderNet}_{\text{L}}$. As will be
shown later, pose refinement can largely improve the pose accuracy, especially
in the strict threshold region.

Figure 3:  The training process of the RenderNet. RetrievalNet and DescNet are
teacher models which take real RGB images as input, while
$\text{RenderNet}_{\text{G}}$ and $\text{RenderNet}_{\text{L}}$, which take
projected colors and local features as input, mimick the outputs of
RetrievalNet and DescNet to generate global features and local features for
virtual views.

###  3.2 RenderNet

Here we present RenderNet to render an arbitrary novel virtual view. We
directly generate the needed features rather than rendering a real image to
avoid the artifacts when generating images. More specifically, RenderNet
contains two branch: 1) a $\text{RenderNet}_{\text{G}}$ to generates a global
feature for image retrieval. 2) a $\text{RenderNet}_{\text{L}}$ branch to
generate a set of local features of keypoints in this view for sparse
matching. For the local branch, instead of re-detecting interest points in the
virtual views, we reuse the projected keypoints from existing views and just
generate revised descriptors that better describe this view.

Input of RenderNet. The input of RenderNet contains densely projected color
and local feature information from nearby real database images, which is a
$H\times W\times 3$ and $\frac{H}{8}\times\frac{W}{8}\times n$ tensor
respectively, where $n$ is the channel dimension. Zeros are padded if no
points are projected. Since local feature descriptors are good representations
of image patches, we expect the encoded patch information can help resolve the
artifacts in the projected color images. We assume the database contains a
coarse 3D model which can provide coarse depths to remove occluded points when
projecting. A pre-trained DescNet is used as the local feature network, which
is SuperPoint in our work.

Output of RenderNet. The $\text{RenderNet}_{\text{G}}$ branch aims to give a
global representation of this virtual view. Here we adopt the widely used
generalized mean pooling (GeM) layer to summarize this view. It outputs a
$m$-d global feature vector which is used for image retrieval. In the
retrieval stage, similarities are measured between real feature vectors and
virtual feature vectors. So in order to ensure the distance comparison
meaningful, we train the $\text{RenderNet}_{\text{G}}$ to minimize the
distance between its output and the feature generated by RetrievalNet which
takes a real image of this view, as follow,

$argmin_{\phi}||\text{RenderNet}_{\text{G}}(I_{p},F_{p})-\text{RetrievalNet}(I)||^{2},$
(1)

where $\phi$ are the parameters of $\text{RenderNet}_{\text{G}}$, $I_{p}$ and
$F_{p}$ are the projected colors images and projected features. $I$ is a real
image of this view.

For the local features of this novel view, we use the projected keypoints and
generate the revised local descriptions using $\text{RenderNet}_{\text{L}}$.
Although the original local descriptors are learned to be invariant to
viewpoint change, we find it is better to revise their representations for the
current novel view, especially when viewpoints change largely. It outputs a
$n$-dimension feature map, then the local features of keypoints are
interpolated from this feature map. Similarly, we train the
$\text{RenderNet}_{\text{L}}$ to minimize the distance of its outputs and the
local features generated by DescNet which takes a real image,

$argmin_{\psi}||(\text{RenderNet}_{\text{L}}(I_{p},F_{p})-\text{DescNet}(I))\cdot
M||^{2},$ (2)

where $\psi$ are the parameters of $\text{RenderNet}_{\text{L}}$. A mask $M$
is used here to only consider the loss around the projected postions.

Network architecture. For real images, we use AP-GeM [revaud2019learning] and
SuperPoint [detone2018superpoint] as the image retrieval and local feature
extraction network respectively. As for our RenderNet, we use a modified
ResNet18 as the backbone, which takes $H\times W\times 3$ as input. As shown
in Fig. 3, RenderNet takes projected color as input and generates
$\frac{1}{8}$ sized features. Then the projected $1/8$ sized local features
are concatenated for further processing. Finally, the global feature and local
feature maps are output by $\text{RenderNet}_{\text{G}}$ and
$\text{RenderNet}_{\text{L}}$ branches respectively. For the
$\text{RenderNet}_{\text{G}}$ branch, after generating $1/32$ sized features,
we use GeM pooling layer and a fully connected layer as the AP-GeM network
[revaud2019learning] to output a $m$d global feature. The
$\text{RenderNet}_{\text{L}}$ branch generate a $1/8$ sized local feature maps
as SuperPoint [detone2018superpoint] using convolutional layesr.

Training process. As shown in Fig. 3, we employ distillation training
paradigms to learn the representation directly from teacher models. Pre-
trained RetrivalNet and DescNet serve as teacher models, which take input of
real RGB images. For student model $\text{RenderNet}_{\text{G}}$ and
$\text{RenderNet}_{\text{L}}$, their inputs are the projected colors and local
features from other views. The output of teacher models serves as targets of
$\text{RenderNet}_{\text{G}}$ and $\text{RenderNet}_{\text{L}}$ respectively.
L2 loss is used in the training. We initialize the weights of head layers of
$\text{RenderNet}_{\text{G}}$ and $\text{RenderNet}_{\text{L}}$ from their
pre-trained teacher models, and fix the weights during training.

##  4 Experiments

###  4.1 Experiment settings

Training data. We use Scannet [dai2017scannet] to train RenderNet. Scannet is
an RGB-D video dataset consisting of 2.5M views in 1513 scenes with camera
poses and dense depth maps provided. For each training image, we uniformly
sample 4 nearby images and generate a virtual view by projecting color and
local features to this view as illustrated in Sec. 3.2. One training sample
consists of a real image and a corresponding projected virtual image.

Testing data. We evaluate our method in the public available dataset InLoc
[taira2018inloc]. InLoc contains 2 large-scale indoor scenes, covering around
10000 $m^{2}$ areas, which are captured using a high-end 3D scanner. It
provides 4680 RGB database images extracted from original RGBD panoramic
images and 356 RGB query images with ground truth pose. We use standard
evaluation metrics accuracy under threshold of ((0.25m, 2°) / (0.5m, 5°) /
(5m, 10°)). Before testing, our RenderNet models are finetuned the Inloc
dataset using their database images.

Implementation details. We train the RenderNet for 30 epochs with a batch size
of 24 on four RTX 2080TI GPUs, 80000 pairs are randomly sampled in each epoch.
When testing, global features are whitened using Principal Component Analysis
(PCA) trained from descriptors of all real and virtual images. Local features
are matched using SuperGlue [sarlin2020superglue], a transformer-based sparse
feature matcher, which is trained with SuperPoint local features. Top 40
database images are matched for the Inloc dataset.

| DUC1 | DUC2
---|---|---
InLoc[taira2018inloc] | 40.9 | 58.1 | 70.2 | 35.9 | 54.2 | 69.5
DensePNV[taira2019right] | 40.9 | 63.6 | 71.7 | 42.7 | 61.8 | 71.0
R2D2[humenberger2020robust] | 41.4 | 60.1 | 73.7 | 47.3 | 67.2 | 73.3
D2Net[dusmanu2019d2] | 42.9 | 63.1 | 75.3 | 40.5 | 61.8 | 77.9
NCNet[rocco2018neighbourhood] | 43.4 | 64.6 | 77.8 | 45.0 | 62.6 | 73.3
HLoc[sarlin2019coarse] | 46.5 | 65.7 | 77.8 | 51.9 | 72.5 | 79.4
SparseNCNet[rocco2020efficient] | 47.0 | 67.2 | 79.8 | 43.5 | 64.9 | 80.2
SuperGlue + Patch2Pixel[zhou2020patch2pix] | 50.0 | 68.2 | 81.8 | 57.3 | 77.9 | 80.2
RLOCS[fan2020visual] | 47.0 | 71.2 | 84.8 | 58.8 | 77.9 | 80.9
LoFTR[sun2021loftr] | 47.5 | 72.2 | 84.8 | 54.2 | 74.8 | 85.5
Ours | 58.6 | 77.8 | 89.4 | 74.8 | 82.4 | 85.5
|  |  |  |  |  |

Table 1: Performance comparison in terms of accuracy under threshold of
(0.25m, 2°) / (0.5m, 5°) / (5m, 10°) in InLoc benchmark.

###  4.2 Results of InLoc

For the inloc dataset, we generate novel virtual views by moving the camera
center horizontally to four positions in the front, back, left, and right
respectively. At each position, 12 images are rendered by rotating the camera
every 30° horizontally. The moving distance is 5m and 2.5m for DUC1 and DUC2
respectively, finally resulting in 5556 valid virtual images. In Fig. 4, we
show the virtual views retrieved by $\text{RenderNet}_{\text{G}}$ for
different query images. The virtual views shown in the figure are generated by
projection from nearby views. It can be identified that the added views have
larger overlaps with query images than the real database image, which makes it
easier to be retrieved and matched. For quantitative evaluation, we compare
RenderNet with various baselines in terms of accuracy under threshold of
(0.25m, 2°) / (0.5m, 5°) / (5m, 10°), such as sparse image matching based
methods [dusmanu2019d2, humenberger2020robust, fan2020visual,
zhou2020patch2pix] and dense image matching based methods
[rocco2018neighbourhood, rocco2020efficient, sun2021loftr]. The results are
shown in Tab. 1, our method gives much better results than the other methods,
especially under the threshold of 0.25m. Compared with very recent state-of-
the-art LoFTR [sun2021loftr] which is a transformer-based dense image matching
method, we achieve improvements on most metrics, which are 11.1%/5.6%/4.6% and
20.6%/7.6%/0.0% on DUC1 and DUC2 respectively.

Figure 4:  Rendered virtual images can have larger overlaps with queries than
existing real database images in the Inloc dataset. Top: query images. Middle:
most similar virtual images. Bottom: most similar real database image.

###  4.3 Ablation Study

| DUC1 | DUC2
---|---|---
Baseline | 50.0 | 69.7 | 82.3 | 48.9 | 68.7 | 73.3
VA (RenderImageNet) | 45.5 | 70.2 | 84.3 | 55.0 | 74.8 | 79.4
VA (RenderNet) | 54.0 | 75.3 | 88.9 | 64.9 | 84.0 | 85.5
VA + PR (Full RenderNet) | 58.6 | 77.8 | 89.4 | 74.8 | 82.4 | 85.5
VA (RenderNet w/o local) | 42.4 | 69.2 | 80.8 | 64.1 | 80.9 | 83.2
VA (RenderNet w/o finetune) | 47.5 | 71.2 | 85.4 | 56.5 | 75.6 | 78.6

Table 2: Ablation about view augmentation (VA) and pose refinement (PR) in the
Inloc dataset. Results are given in terms of accuracy under threshold of
(0.25m, 2°) / (0.5m, 5°) / (5m, 10°).

View augmentation. We first show the effectiveness of view augmentation. Our
baseline is a modified HLoc [sarlin2019coarse], which uses DIR
[revaud2019learning] network for image retrieval. This baseline achieves
higher results than the original HLoc in DUC1. In Table. 2, the notation VA
means that we use view augmentation in the localization pipeline.

As is shown, view augmentation via RenderNet in Tab. 2) can largely boost the
performance, achieving an improvement of 4.0%/5.6%/6.5% and 16.0%/15.3%/12.2%
in DUC1 and DUC2 respectively. In order to compare with the view synthesis
method which generates a real image for localization, we design a
RenderImageNet which also takes projected image and local features as input.
The RenderImageNet is learned to synthesize a real image of the input view
similar to our RenderNet. It is also trained on Scannet and finetuned on Inloc
dataset. As shown in the Table. 2, our RenderNet can achieves much better than
VA (RenderImageNet in Tab. 2), 8.5%/5.1%/4.6% and 9.9%/9.2%6.1% higher on DUC1
nad DUC2 respectively. This demonstrates that directly rendering the needed
features can be more robust than synthesizing images. Compared with DUC1, the
improvement of RenderNet in DUC2 is higher. This is because DUC2 is more
narrow and scene appearance changes larger when viewpoint changes, thus
rendering additional views is more helpful for DUC2.

Effectiveness of $\text{RenderNet}_{\text{L}}$. In order to verify the
effectiveness of $\text{RenderNet}_{\text{L}}$, we directly use the projected
local feature descriptors without revising the features for the current view.
Although SuperPoint are learned to be invariant to view changes, the revised
features learned by our $\text{RenderNet}_{\text{L}}$ can achieve higher
results in large viewpoint change case, with an improvement of 8.1% and 2.3%
under the 5m, 10° threshold, comparing to the VA + RenderNet w/o local shown
in Table. 2.

Effectiveness of finetune. In Table . 2, we also report the results without
finetuning RenderNet on the target dataset Inloc. It can also improve the
localization performance. After finetune, RenderNet can further achieve 3.5%
and 6.9% higher results under the 5m, 10° threshold. Since the finetune can be
applied offline and does not increase the cost of online relocalization
system, we recommend a finetune on the target dataset for better performance.

Pose refinement. We then validate the effectiveness of pose refinement. In
Table. 2, Full RenderNet means we adopt both view augmentation and pose
refinement. Using both rendered virtual images and real images can largely
improve the pose accuracy. As is shown, when using pose refinement, the
accuracy under 0.25m, 2° can be further increased by 4.6% and 9.9% compared to
only using view augmentation. We can see that view augmentation improves all
the metrics while pose refinement mainly increases the accuracy of threshold
0.25m by a large margin. The reason is that view augmentation makes it more
likely for query images to retrieve views with large overlaps. On the other
hand, pose refinement fine-tunes the coarse poses to a more accurate one.
Combining these two approaches, the RenderNet can boost the performance by
8.6%/8.1%/7.1% and 25.9%/13.7%/12.2%.

##  5 Conclusion

In the work, we propose RenderNet for relocalization in large scale indoor
environments, which generates virtual views for view augmentation and pose
refinement without requiring high-quality 3D models. It consists of
$\text{RenderNet}_{\text{G}}$ to generate global features and
$\text{RenderNet}_{\text{L}}$ to generate local descriptors for virtual novel
views, which are then used in the relocalization pipeline for image retrieval
and image matching. These added novel views help retrieve the right place and
establish more feature correspondences. We show in experiments that our method
can largely improve the relocalization performance in the large-scale indoor
dataset Inloc.

## Checklist

The checklist follows the references. Please read the checklist guidelines
carefully for information on how to answer these questions. For each question,
change the default to , , or . You are strongly encouraged to include a
justification to your answer, either by referencing the appropriate section of
your paper or providing a brief inline description. For example:

  * •

Did you include the license to the code and datasets? See Section
LABEL:gen_inst.

  * •

Did you include the license to the code and datasets? The code and the data
are proprietary.

  * •

Did you include the license to the code and datasets?

Please do not modify the questions and only use the provided macros for your
answers. Note that the Checklist section does not count towards the page
limit. In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope?

    2. (b)

Did you describe the limitations of your work? . See section
LABEL:sec:limitation.

    3. (c)

Did you discuss any potential negative societal impacts of your work? . See
section LABEL:sec:limitation.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them?

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results?

    2. (b)

Did you include complete proofs of all theoretical results?

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? . Code
will be released upon acceptance.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? . Dataset and training setting can be found in section
4.1. More testing setting for Inloc and RIO can be found in section 4.2 and
LABEL:sec:exp_rio.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? . Error bars are not reported because it would be
too computationally expensive

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? . Only one GPU is
used. See section 4.1.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? . We use Scannet
[dai2017scannet], Inloc [taira2018inloc] and RIO [wald2020beyond] datasets.
Code of SuperPoint [detone2018superpoint], SuperGlue [sarlin2020superglue],
HLoc [sarlin2019coarse], DIR [revaud2019learning] and Disk
[tyszkiewicz2020disk] are used.

    2. (b)

Did you mention the license of the assets? . But the licenses of datasets and
code can be found in their website or Github repositories.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? . We do no use new assets.

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? . According to the license, we can use them for
research purpose.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? .

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable?

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable?

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation?


§ VELOCITY DISPERSION

Panel a. Dispersions in $U$, $V$, and $W$ calculated for samples all_100, red_100, and blue_100 using a moving window in absolute magnitude and shown with black, red, and blue colours respectively. The shaded region shows the 1$\sigma$ uncertainty. The red and blue lines appear to be separate for brighter white dwarfs in all three directions but become mixed towards the fainter end of the sequence. Panel b. Same as a. but for samples all_200, red_200, and blue_200. For these WDs the split between the red and the blue sequences is much more pronounced and now clearly so at all absolute magnitudes. The red sequence appears to have a larger velocity dispersion in all directions and at almost all absolute magnitudes.

Cumulative distribution of the $Q$-statistic described in the text for the red and blue sequences when comparing samples limited to within 100 pc or between 100-200 pc. The black line shows the Gaussian CDF with $\mu=0$ and $\sigma=1$.

The velocity distribution of WDs in $U$ and $V$. The top row shows the distribution for the whole WD sequence and the red and blue sequences corresponding to samples all_500, red_500, and blue_500 in Table <ref>. Contour lines are constructed so as to contain 95, 90, 80, 68, 50, 33, 21, 12, 6, and 2 percent of all sources in the sample and magenta crosses show identified peaks with a peak-finding algorithm described in Section <ref>. The second row contains samples A, B, and C as indicated which are magnitude bins in the full sample. Three horizontal arches are illustrated in sample C and explained in section <ref>. The third row takes the distributions for the three samples A, B, and C and subtracts the mean of all three samples. The contour lines are the same as the individual distributions. The fourth row shows the first 9 groups identified in <cit.> as black crosses for comparison.

Same as Fig. <ref> but in $U$ and $W$. The circles in panel C marks the peaks that are suggested to be part of a double-peaked feature discussed in section <ref>

Same as Fig. <ref> but in $V$ and $W$.

In section <ref> we showed how we can calculate the velocity dispersion of a sample of stars using only their positions and tangential velocities. We calculate 3D velocity dispersions for a moving window in absolute magnitude separately for the WD samples limited to 100 pc and 200 pc. We estimate the uncertainty within the windowed sample by calculating the moments for 500 bootstrapped samples. The standard deviation of all of the resulting statistics is used to provide a 1$\sigma$ uncertainty. The complete 100 pc sample's velocity dispersions are seen in panel a. of Fig. <ref>. Generally we would expect the velocity dispersion to increase with fainter magnitude due to dynamical heating over time. This is true overall, though the component $\sigma_V$ is approximately constant for the red and black sequences over the range of magnitudes considered.

The red sequence has a larger velocity dispersion than the blue at low magnitudes in all directions to the extent that their 1$\sigma$ regions do not overlap. Beyond this the separation is not as great and harder to make out. In both $U$ and $V$ the median places the red sequence above the blue. For a comparison we look also at the velocity dispersion for all the WDs within 200 pc in panel b. in which the separation is very clear at all magnitudes and for all three velocity components. The 1$\sigma$ regions of the red and blue barely overlap with the joint sample. To determine whether or not the two distance-limited samples probe the same underlying distribution, we calculate the statistic

\begin{equation}
Q = \frac{\sigma_{A} - \sigma_{B}}{\sqrt{\Delta\sigma_{A}^2 + \Delta\sigma_{B}^2}},
\end{equation}

where $\sigma_{A,B}$ is the velocity dispersion for samples A or B and $\Delta\sigma_{A,B}$ their uncertainties estimated through bootstrapping. We calculate Q for both sequences with A being the 100 pc limited samples and B selected to be the stars between 100-200 pc to avoid overlapping sources. We calculate Q for a number of binned points in $M_G$ and if the two distance-limited samples are drawn from similar underlying distributions the distribution of $Q$ should be Gaussian with $\mu=0$ and $\sigma=1$. The cumulative distribution of $Q$ can be seen in Fig. <ref> for the red and blue sequences and shows good alignment with the analytical Gaussian CDF. In addition to this we perform a Kolmogorov-Smirnov (KS) test between the different samples and the Gaussian CDF under the null hypothesis that they arise from the same distribution. The KS testing yields a $p$-value of 0.936, 0.834, and 0.936 when testing the red/blue, red/Gaussian, and blue/Gaussian Q distributions respectively, clearly demonstrating that they arise from the same Gaussian distribution and therefore the difference between our two samples is simply due to statistical noise. In summary, there is a statistically significant separation in the velocity dispersion between the red and blue sequences.

If the bifurcation is caused by a bimodal WD mass distribution, this could be explained by the progenitors of the WDs in the two different sequences. The lower mass WDs will come from less massive main-sequence stars, which would mean that these progenitors will have been dynamically heated for a longer duration of time and the lower mass WDs would be born with larger dispersions.

The separation between red and blue sequences is larger at brighter magnitudes, which corresponds to younger WDs. This can be expected if the blue sequence comes from massive WDs which are younger than their red sequence, less massive, counterparts. Most of the dynamical heating occurs during the first few Gyr of a stars life (e. g., ) and so would already have occurred for the red sequence WDs but still be occurring for the blue sequence ones. Over time as WDs on both sequences cool to fainter magnitudes their heating tracks would align to become more parallel, which is visible in our figures.

Undeniably the bifurcation in the CMD is well-described by a difference in the atmospheric composition of WDs <cit.>. It is however difficult to reconcile with the observed difference in the kinematics that we display here which requires significantly different dynamical heating histories between the two sequences. If there is a significant age difference between the two populations however this would result from the expected heating from the Galactic disk. We can estimate the age difference of stars in the two sequences using the age-velocity dispersion relationship from <cit.> and applying it to the total velocity dispersion of the two samples,  and  . This suggests a typical age of  $\sim$7.7 Gyr and $\sim$4.9 Gyr respectively for the red and blue sequences. We are unaware of any mechanism that could explain these kinematics purely by a difference in atmospheric composition.

As mentioned in Section <ref> we also selected WDs out to $\varpi = 2$ mas or 500 pc. Beyond 200 pc the separation between red and blue sequences remains the same and did not provide any further insights.

§ VELOCITY DISTRIBUTION OF WDS

We apply the maximum penalized likelihood estimate to all of the samples to produce a full 3D velocity distribution for which we show the projections in the planes $U$-$V$, $U$-$W$, and $V$-$W$. To identify where the peaks of the distribution are we apply the peak_local_max function available as part of the scikit-image[<https://scikit-image.org/>] Python package <cit.>. The results in the $U$-$V$ plane are presented first in Fig. <ref>. In the first row we compare the velocity distribution of all our WDs with those of WDs identified as belonging to the red and blue sequences as laid out in Section <ref>. It is immediately clear that the WD velocity distribution is overall very similar to the rest of the Solar Neighbourhood (e.g., ) which is not too surprising considering that the WDs are older stars of the same population.

We can identify a few familiar moving groups such as the Hyades, Pleiades, and Hercules across all three samples very clearly. To a lesser extent we can also identify higher velocity moving groups beyond $U > 25$ km s$^{-1}$ such as Wolf 630 and Dehnen98.

Identified features in the velocity distributions at approximate coordinates. We omit the velocity in $W$ when it cannot be determined accurately. Filled or hollow circles are for clear and weak features respectively. Group 5 has an asterisk indicating unclear membership in the moving group which is further discussed in Section <ref>.

Feature     $U$     $V$     $W$      all_500     red_500     blue_500     A     B     C     Moving Group
1    4         -1             …                                                  Sirius

2    -13       -11        -10                                                     Coma Berenices

3    -28       -17       …                             Hyades

4    -28       -47      …                            Hercules

5    7        -19          …                                                                                Coma Berenices$^\ast$

6    1         -48         …                                              Hercules

7    1        -15         …                                                                                Coma Berenices

8    16        -26        …                                                                                Wolf 630

9    -19       -23       -8                                    Pleiades

10    40        -27      …                            Dehnen 98

11    19        -1           …                                                                                Sirius

12    -7        -5           …                                                                                Coma Berenices

The moving groups in the blue sequence are more narrowly distributed around their group mean overall than those of the red sequence, where the groups are spread out more across the central regions in the space due to the overall larger velocity dispersion shown in Section <ref>.

We show the three samples of different absolute magnitudes in the second row of Fig. <ref>, going from brightest (A) to faintest (B). For the individual distributions we see the same type of structure as we did for the whole sample. It appears as though there is more structure in these subsamples than the former three. However, as the samples contain less stars than the all_100 sample but use the same smoothing parameter $\alpha$ it has a noisier distribution. The fainter sample, C, has older stars and thus a larger velocity dispersion as expected which `smears' out the distribution which reveals an arch-like structure in C, with three horizontal arches at $V$ of 0 km s$^{-1}$, -20 km s$^{-1}$, and at -40 km s$^{-1}$ which are illustrated in the plot. This type of structure has been shown to exist in the Solar Neighbourhood <cit.> and can be attributed to dynamical resonances with the spiral arms and Galactic bar <cit.>.

The third row takes the three samples A, B, and C, and subtracts the mean of all three to emphasise the parts of the distribution that are shared between the samples or are independently significant. Our expectation is that the bright, young sample with low velocity dispersion (A) should be over-abundant near the centre of the distribution and the faint sample should show the opposite. Stars with very large $|V|$ belong to populations with their guiding centres at significantly larger/smaller Galactocentric radii and would be visiting the Solar Neighbourhood when their orbits are sufficiently dynamically heated. Thus, they would be older and part of a fainter sample. This effect can be seen in the third row where samples A and B dominate the centre of the distribution whilst C is represented more in the wings with, for example, the Hercules moving groups being overabundant. Asymmetric drift also causes the mode of the distribution to shift towards more negative $V$.

The region around $(U, V) \approx (7, -19)$ km s$^{-1}$ is underdense in A and B but surprisingly is over-abundant in C, with an identified peak in that region as well. A review of recent articles that investigate the velocity distribution in $U$-$V$ for the Solar Neighbourhood reveals that <cit.> and <cit.> do not identify a known moving group in this region while <cit.> does and attributes it to the Coma Berenices stream.

The distributions in $U$-$W$ and $V$-$W$ in Fig. <ref> and Fig. <ref> are not as rich in structure as the $U$-$V$ plane, but we can, however, identify Coma Berenices and Pleiades by comparing the two planes. Our choice of smoothing parameter $\alpha$ is chosen to fit best with the sample all_500 and hence will be under-estimated for the smaller samples (red_500 and blue_500) and as such the features seen in them may disappear with an $\alpha$ that is appropriate for smaller samples. A feature identified by <cit.> is a double-peaked feature along $W$ in the $U$-$W$ plane at $U \approx -30$ km s$^{-1}$ and in the $V$-$W$ plane at $V \approx -20$ km s$^{-1}$. This double-peaked feature is very vaguely present in $U$-$W$ for sample C. In the $V$-$W$ plane however we can identify the feature clearly in both red_500 as well as blue_500, and perhaps vaguely in B and C. We mark the proposed double-peak feature with circles around the involved features. This seems to imply that the double-peaked feature, of which the Pleiades appears a part of, is limited to fainter, possibly older stars.

In Table <ref> we summarise the features we can see and mark whether we can see them clearly, somewhat, or not at all.  We also list their locations in all three velocity dimensions (see for comparison Table 2 from ). For most of the moving groups we are able to associate a previously known one following <cit.> but no known group accurately describes feature 5 in the table, meaning that it may be a new feature. Compared to previous results the features presented here are located mostly at lower velocities ($|v| < 50$ km s$^{-1}$). It may be that the WDs in groups at larger velocities are not numerous enough to appear in the distributions or that their dispersion is too great.

§.§ The stellar warp

Mean of the vertical motion, $\langle W\rangle$, in the central regions of $f(V, W)$ for sample all_500. The location of the LSR is marked by a triangle and is taken from <cit.>. The errorbars show the $1\sigma$ Poisson noise. The dashed black line shows a weighted linear fit.

In his analysis of Hipparcos data, <cit.> found evidence for the stellar warp by investigating $\langle W\rangle$ as a function of $V$. For $V > 10$ km s$^{-1}$ the velocity distribution was skewed towards positive $\langle W\rangle$. We perform the same analysis here of our largest WD sample all_500 and the results can be seen in Fig. <ref>.  To do this we require a larger grid of $\pmb{n} = [200, 152, 152]$ covering $U \in [-200, 200]$ km s$^{-1}$, $V \in [-150, 150]$ km s$^{-1}$, and $W \in [-150, 150]$ km s$^{-1}$ to ensure that none of the edge effects discussed in Section <ref> affect the results. We find that our results also show increasing $\langle W\rangle$ with increasing $V$.

We compare our results to <cit.> who use Gaia-TGAS to determine the average vertical velocity as a function of azimuthal velocity and angular momentum for stars in the two cones towards the Galactic centre and anti-centre with angular radius of 30$^\circ$ (their Fig. 6). Since our samples are in the Solar Neighbourhood the velocity in $V$ is a proxy for angular momentum since $L_z = -R(V_\odot + V)$. Just as in our sample, the average vertical velocity increases with angular momentum and shows a dip just beyond the Local Standard of Rest (LSR). At large angular momentum, corresponding to $V > 15$ km s$^{-1}$ in the Solar Neighbourhood there a significant increase in $\langle W\rangle$ for all of their samples as well as ours. Towards  $V = 50$ km s$^{-1}$ $\langle W\rangle$ decreases again which is consistent with a similar decrease at higher angular momentum seen by <cit.>.

More recently, <cit.> also looked at the vertical velocity profile of stars outside the solar radius as a function of angular momentum. We see this in their fig. 11 which shows this profile for several different stellar types, of which the young main-sequence stars are the ones to show a decrease in vertical velocity at $L_z^\ast \sim 2400$ kpc km s$^{-1}$ corresponding to around $V = 60$ km s$^{-1}$. However, our sample and the Gaia-TGAS sample are in the Solar Neighbourhood where stars with such large angular momentum must have large radial excursions to be included in the sample, and therefore are very likely to be old. This is in contrast to the stars in <cit.> which are young and located beyond 10 kpc from the Galactic centre. While the decrease in the vertical velocity profile we see could therefore be related to these decreases seen in other datasets, we note that the decrease is still within 1$\sigma$ uncertainty, and therefore the most we can say is that the results are mutually consistent.

For $V < 20$ km s$^{-1}$ where the Poisson noise is not as great, the average velocity, $\langle W\rangle$, has a relatively small tilt. We perform a weighted linear fit for $V \in [-80, 50]$ km s$^{-1}$ and find a slope of $\sim$0.03 (shown in Fig. <ref>), which is somewhat larger than $\sim$0.02 found by <cit.>. If we instead restrict our fit to the relatively well constrained region $V \in [-80, 20]$ km s$^{-1}$ we find a slope of $\sim$0.025 in agreement with the previous results.