# A Lightweight Domain Adaptive Absolute Pose Regressor Using Barlow Twins
Objective

Praveen Kumar Rajendran1 Quoc-Vinh Lai-Dang1 Luiz Felipe Vecchietti2 Dongsoo
Har1
1KAIST, Daejeon, South Korea 2IBS, Daejeon, South Korea
{praveenkumar, ldqvinh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Identifying the camera pose for a given image is a challenging problem with
applications in robotics, autonomous vehicles, and augmented/virtual reality.
Lately, learning-based methods have shown to be effective for absolute camera
pose estimation. However, these methods are not accurate when generalizing to
different domains. In this paper, a domain adaptive training framework for
absolute pose regression is introduced. In the proposed framework, the scene
image is augmented for different domains by using generative methods to train
parallel branches using Barlow Twins objective. The parallel branches leverage
a lightweight CNN-based absolute pose regressor architecture. Further, the
efficacy of incorporating spatial and channel-wise attention in the regression
head for rotation prediction is investigated. Our method is evaluated with two
datasets, Cambridge landmarks and 7Scenes. The results demonstrate that, even
with using roughly 24 times fewer FLOPs, 12 times fewer activations, and 5
times fewer parameters than MS-Transformer, our approach outperforms all the
CNN-based architectures and achieves performance comparable to transformer-
based architectures. Our method ranks $2^{nd}$ and $4^{th}$ with the Cambridge
Landmarks and 7Scenes datasets, respectively. In addition, for augmented
domains not encountered during training, our approach significantly
outperforms the MS-transformer. Furthermore, it is shown that our domain
adaptive framework achieves better performance than the single branch model
trained with the identical CNN backbone with all instances of the unseen
distribution.

##  1 Introduction

Absolute camera pose estimation, e.g., predicting the 3-Dimensional position
and orientation of a camera for a given image, is a challenging problem with
numerous practical applications in computer vision, ranging from mobile robots
[1] and autonomous vehicles [8, 26] to augmented/virtual reality [10].
Traditional camera pose estimation techniques such as structure-from-
motion(SfM) enabled by perspective-n-point(PnP) and RANSAC [16] tend to
exploit multi-view geometry scene constraints, using 2D-3D correspondences for
predictions [18]. The SfM-based methods are usually dependent on image
descriptors such as SIFT[33], ORB[52], and FAST[44]. However, these
descriptors suffer poor performance in handling scenes with abrupt changes of
illumination, occlusions, or repeated structures. In addition, the
computational cost is significantly high. Another alternative solution to
tackle this problem is image retrieval (IR) methods [66, 2, 53, 63, 42, 55].
For the IR methods such as NetVLAD [2], similar images to the query image are
searched in the dataset and are used for feature extraction, matching, and
pose estimation. However, the IR methods take relatively large running time.

Learning-based approaches have shown credible results for a variety of tasks
[28, 70, 69, 39]. Thus, to tackle the shortcomings of traditional camera pose
estimation methods, lately, end-to-end pose regressors involving deep learning
techniques have been investigated. The first absolute pose regressor(APR),
PoseNet [25], predicts the camera pose for a given image using an end-to-end
model consisting of GoogLeNet [62] convolutional neural network (CNN)
backbone. Following the PoseNet [25], multiple research works [23, 24, 71, 9,
40, 36, 7, 5, 72, 56, 57, 58] investigating end-to-end methods have been
proposed. Given their fixed architecture, these methods have a fixed memory
footprint regardless of the size of the scene and their running time is an
order of magnitude faster than traditional structure-based methods. However,
the accuracy of these methods is still not superior to structure-based methods
that make use of 3D information during inference.

Typically, the aforementioned end-to-end methods are trained in a supervised
fashion using a dataset consisting of images and ground truth poses for a
single scene. Because of that, using the same trained model for images
collected in different scenes is non-trivial. Recently, methods that are
trained using a dataset with data obtained from multiple scenes have been
proposed to alleviate this problem. Multi-Scene PoseNet (MSPN) [5] and MS-
Transformer [57] train a model with multiple scene data. These absolute pose
regressors (APRs) based on multiple scenes aim to learn general embeddings for
different scenes when compared to scene-specific APRs. During inference, it is
observed that multiple scene APRs increase the robustness when tested with
unseen scene images. To improve the performance and robustness of end-to-end
camera pose estimation, methods that use as input several images captured
sequentially have also been investigated [7, 12].

Scene-specific APRs are not designed to solve domain invariance by nature,
thus they lack the ability to generalize well to different domains of the same
scene. Despite being trained with multiple scene data, the MSPN and MS-
Transformer also do not encourage domain invariance to different domains.
Unlike these approaches, our method aims to address the domain invariance of
APRs. In this paper, we hypothesize that using image pairs for a given pose
under different conditions as input and using an objective for domain
invariance should improve the accuracy of a given APR in unseen domains when
compared to the APR that is trained only in the real distribution. To check
our hypothesis, we introduce a robust domain adaptive framework for absolute
pose regression. In our work, scene images are augmented for different domains
by using generative methods to train parallel branches leveraging a
contrastive Barlow Twins objective inspired by [75]. This objective has the
goal of reducing the difference of embeddings for images taken from the same
pose under different domains. The proposed training framework is general and
can be applied to different CNN-based APRs for improving domain invariance.
Additionally, the number of parallel branches used for training can be
adjusted for different tasks. The proposed training framework is shown in Fig.
1(a) for three parallel branches with shared weights, one for the original
image and two for domain augmentations. Domain augmentations are performed by
processing the original image using generative adversarial network (GAN)
methods such as ManiFest [46] and CoMoGAN [45]. While the proposed framework
ensures the robustness of the model by using parallel branches during
training, for inference, as shown in Fig. 1(b), the model is loaded as the
single branch since parallel branches share weights thus not adding
computational complexity when compared to other methods.

For our study, we focus on using an APR with a lightweight CNN-based
architecture for embedded applications. The pretrained MobileNetV3-Large [20]
backbone is leveraged for learning domain invariant embeddings. These
embeddings are then used by a regression head to predict the camera pose,
e.g., translation and rotation vectors. For the rotation vector prediction,
adding spatial and channel-wise attention to focus on important features
related to a specific view of the scene is investigated. The proposed training
framework as well as APR architectures are evaluated by using two datasets,
the 7Scenes dataset [17], which contains multiple indoor scenes, and the
Cambridge Landmarks dataset [25], which contains multiple outdoor scenes. The
results show that our method outperforms CNN-based architectures while
achieving comparable performance to transformer-based architectures. It is
noted that our method is highly efficient regarding FLOPs, number of
parameters, and memory requirements when compared to baseline methods.
Ablation studies are performed to investigate the importance of each of the
proposed components to the final performance. Additionally, we evaluate our
method in three domains unseen during training and show that it achieves
superior performance when compared to baseline methods. Throughout this work,
we use the term $\mathcal{DA}$-model to refer to the proposed Domain Adaptive
training framework while $\mathcal{SB}$-model is used to refer to a Single
Branch APR training framework. Our main contributions can be outlined as
follows.

Figure 1: Overview of the proposed framework. (a) Domain adaptive training
framework. Three parallel branches with shared weights are trained for images
related to the same pose under different domains. (b) Inference framework.

  * •

We propose a domain adaptive training framework for absolute pose regressors
by using a contrastive Barlow Twins objective along with an L2 loss to train
parallel branches. The Barlow Twins loss enforces invariance and redundancy
reduction on learned embeddings for the same pose images under different
domains. The framework is general and can be applied to different absolute
pose regressors.

  * •

Shared weights are maintained in parallel branches during training, thus,
during inference, our model can be loaded as a single branch model keeping the
same runtime complexity when compared to CNN-based absolute pose regressors.

  * •

The addition of spatial and channel-wise attention to the regression head
responsible for rotation prediction is investigated. The attention layer
incorporated improves rotation prediction in our framework while introducing
only about 11K additional parameters($\sim$ 0.1MB) to the network.

  * •

We propose the use of a lightweight MobileNetV3-Large backbone for the
experiments using the proposed domain adaptive framework. It is shown that,
even with the use of a lightweight architecture, our method outperforms other
CNN-based baseline methods and achieves comparable performance to transformer-
based architectures in domains seen during training while outperforming all
baseline methods on domains unseen during training.

##  2 Related Work

In this section, end-to-end deep learning methods for camera pose estimation
are reviewed. Additional related work regarding structure-based localization
and image retrieval methods is presented in Appendix H.

Relative Pose Estimation: Deep learning-based regression is not only being
used in absolute pose regression tasks but also adopted in another body of
work, the relative camera pose estimation investigated by RPNet [14] and
others [37, 32, 73, 48]. In a relative pose estimation problem, the input
given is a pair of images with different poses, and the output is the relative
transformation (translation and rotation vectors) between the poses
represented by the input images. To tackle this problem, Melekhov et al[37,
32] investigated relative pose estimation with end-to-end deep learning
methods. The RPNet [14] is proposed as an end-to-end approach to obtain the
full translation vector while previous approaches obtained a translation
vector up to a scale factor. Following RPNet, the RCPNet [73] used a ResNet-
based model with a learnable pose loss objective. The RelMobNet [48] involved
the use of a lightweight CNN backbone and different training strategies to
improve computational efficiency for relative pose estimation. Inspired by the
methods that predict keypoint locations in a scene [60, 61, 31, 34],
DirectionNet[11] proposed the prediction of a discrete distribution over
camera poses and demonstrated promising results. Recent research [51]
investigated Vision Transformers (ViT) to estimate the relative pose.

Absolute Pose Estimation: In this subsection, we review methods that,
similarly to the method proposed in this work, automatically learn features
for absolute pose estimation. We define these methods as absolute pose
regressors (APRs). The end-to-end APR using a CNN to estimate a 6 Degree of
Freedom (DoF) camera pose was PoseNet [25]. The model is trained on a dataset
of a single landmark scene. The regression objective is defined to minimize
the deviation for translation and rotation vectors, in which a hyperparameter
is used to balance between the translation and rotation losses. Geometric
PoseNet [24] increases the robustness of the PoseNet by learning how to
balance the losses automatically as part of the training process. The impact
of geometric reprojection error-based objective is also investigated in [24].

Following the PoseNet, the end-to-end camera pose estimation is further
investigated by combining CNNs and RNNs [71]. Specifically, Walsh et al [71]
apply long-short term memories on top of the learned CNN features for
structured dimensionality reduction to enhance localization performance.
Bayesian PoseNet [23] improves localization accuracy by using a Bayesian CNN
to predict the uncertainty of the model during estimation. GPoseNet [9] uses a
Gaussian Process regressor to learn the probability distribution of the 6DoF
camera position at different poses. SVS-Pose [40] is a method for data
augmentation in 3D space for better pose coverage, which further improves
camera pose estimation. MapNet [7] combines images with multimodal sensory
inputs such as visual odometry and GPS for camera localization. However, it
should be noted that the MapNet relies on the use of multiple sequential
images to predict the final pose. IPRNet [56] exploits the use of general
features obtained by models trained for visual similarity to predict the
camera pose. AtLoc [72] exploits an attention mechanism on top of CNN to focus
on informative features to predict the camera pose. MSPN [5] suggests that a
single model can be used for multiple scenes instead of scene-specific models.
MS-Transformer[57] presents a transformer-based architecture for multiple
scenes, providing state-of-the-art results. The prediction of translation
vectors in MS-Tranformer is further improved by adding a camera pose encoder
and optimization steps as presented in [58].

Figure 2: Left: $\mathcal{DA}$-model learns domain invariant features by using
a training objective composed of a $\mathsf{L}_{2}$ loss and a Barlow Twins
$\mathsf{L}_{\mathcal{BT}}$ loss. Additionally, a pose loss
$\mathsf{L}_{\mathcal{P}}$ is applied to optimize pose predictions. Right: The
inference stage is performed as a single branch model as three parallel
branches share network weights during training.

##  3 Proposed Method

The proposed method is based on two objectives: (i) creation of a domain
adaptive framework that can leverage the use of images corresponding to the
same pose under different domains during training; (ii) use of the proposed
framework with a lightweight APR architecture while keeping competitive
performance with baseline methods. For (i), the use of Barlow Twins objective
[75] is proposed to reduce the difference of embeddings under different
domains. For (ii), the use of a MobileNetV3-Large [20] backbone and the
addition of a multi-head attention(MHA) [68] module are exploited to improve
rotation prediction.

Domain Adaptive ($\mathcal{DA}$) APR Framework: The proposed framework is
general and is to be used with APR architectures. Unlike previous APR
approaches, the framework differs by incorporating a domain adaptive objective
into the model training. This objective requires the availability of multiple
images of the same pose under different domains (light, texture). In this
work, the number $N$ of multiple images obtained for each pose is set to 3. To
obtain these images, we use two methods based on GANs: ManiFest [46] and
CoMoGAN [45]. The ManiFest [46] is a few-shot learning-based image translation
GAN that learns a context-aware representation of a target domain, and, in
this work, is used to generate augmented foggy-view images. The output image
after using the ManiFest is defined as $\mathcal{I}_{fog}$ as shown in Eq.
(1). The CoMoGAN [45] is a continuous image translation method used for time-
lapse generation, and, in this work, is used to generate augmented night-view
images. The output image after using the CoMoGAN is defined as
$\mathcal{I}_{night}$ as shown in Eq. (2).

$\displaystyle\mathcal{I}_{fog}=ManiFest(\mathcal{I}_{real}),$ (1)
$\displaystyle\mathcal{I}_{night}=CoMoGAN(\mathcal{I}_{real}),$ (2)

where $\mathcal{I}_{real}$ is the real image sample contained in the dataset
for a given pose $\mathcal{P}_{GT}\thinspace\in\thinspace\mathbb{R}^{7}$. The
pose $\mathcal{P}_{GT}$ is given as $\mathcal{P}_{GT}=[t,q]$, where
$t\thinspace\in\thinspace\mathbb{R}^{3}$ is the 3-Dimensional translation
vector and $q\thinspace\in\thinspace\mathbb{R}^{4}$ is a 4-Dimensional
quarternion representing the rotation vector. After this preprocessing step,
we have $N=3$ images for the same pose $\mathcal{P}_{GT}$ under different
domains (real, foggy, night). An APR receives an image as input and provides
the predicted pose $\widehat{\mathcal{P}}$ as output for the target scene.
During training, we have $N=3$ parallel branches that provide as outputs
$\widehat{\mathcal{P}_{real}}$, $\widehat{\mathcal{P}_{fog}}$, and
$\widehat{\mathcal{P}_{night}}$ as given in Eqs. (3), (4), and (5) as
following

$\displaystyle\widehat{\mathcal{P}_{real}}=APR(\mathcal{I}_{real}),$ (3)
$\displaystyle\widehat{\mathcal{P}_{fog}}=APR(\mathcal{I}_{fog}),$ (4)
$\displaystyle\widehat{\mathcal{P}_{night}}=APR(\mathcal{I}_{night}).$ (5)

The parallel branches share the same architecture and network weights. The
detailed visualization of the APR architecture utilized in this work for the
prediction of $\widehat{\mathcal{P}_{real}}$, $\widehat{\mathcal{P}_{fog}}$,
and $\widehat{\mathcal{P}_{night}}$ is shown in Fig. 2. The three parallel
branches using MobileNetV3-Large CNN backbones are presented. Details
regarding MobileNetV3-Large architecture is given in Appendix A. The
MobileNet-V3-Large CNN backbones encode the three input images into three
latent feature maps ($\mathcal{F}_{real}$, $\mathcal{F}_{fog}$,
$\mathcal{F}_{night}$) $\in\thinspace\mathbb{R}^{960\times 7\times 7}$. These
feature maps are used to obtain two latent vectors, one for translation and
one for rotation, for each input image, via Global Average Pooling(GAP) and
multi-layer perceptron(MLP). These latent vectors
$\in\thinspace\mathbb{R}^{D}$ are defined as ${latent^{T}}_{real}$,
${latent^{T}}_{fog}$, ${latent^{T}}_{night}$, ${latent^{R}}_{real}$,
${latent^{R}}_{fog}$, and ${latent^{R}}_{night}$, where ${latent^{T}}$ is the
latent vector used to predict the translation vector and ${latent^{R}}$ is the
latent vector used to predict the rotation vector. It is important to note
that, given a variable number of domains for the same pose, this methodology
can be expanded for training with additional parallel branches or for creating
contrastive image pairs for training.

For the translation vector, the latent vector ${latent^{T}}$ of each subgroup
(real, foggy, night) is the input for a MLP that finally predicts the
translation vector $t$. For the rotation vector, the latent vector
${latent^{R}}$ of each subgroup is combined with the features obtained by the
MobileNet-V3-Large CNN backbone to be passed in a multi-head attention layer
as follows. The latent vector ${latent^{R}}$ is passed to an MLP layer to get
the Value $V\in\thinspace\mathbb{R}^{49}$ of the MHA layer. Query
$Q\in\thinspace\mathbb{R}^{49}$ and Key $K\in\thinspace\mathbb{R}^{49}$
vectors are obtained by applying spatial-point-wise max pooling (SMP) and
spatial-point-wise average pooling (SAP) to the feature maps
($\mathcal{F}_{real}$, $\mathcal{F}_{fog}$, $\mathcal{F}_{night}$),
respectively. The intuition behind this step is related to restoring features
that are important to rotation prediction in a given scene. It is important to
mention that the addition of the MHA layer using this formulation introduces
approximately 11k new parameters to the APR architecture. The attention score
from the MHA layer is then used to predict the rotation vector represented as
the quaternion $q$.

Training Objective: As shown in Fig. 2, ${latent^{T}}_{real}$ and
${latent^{R}}_{real}$ are the latent embeddings of $\mathcal{I}_{real}$ image.
The ${latent^{T}}_{fog}$ and ${latent^{R}}_{fog}$ are the latent embeddings of
$\mathcal{I}_{fog}$ image, and, ${latent^{T}}_{night}$ and
${latent^{R}}_{night}$ are the latent embeddings of $\mathcal{I}_{night}$
image. For improved domain invariance, our objective is to increase similarity
among ${latent^{T}}_{real}$, ${latent^{T}}_{fog}$, and ${latent^{T}}_{night}$,
and to increase similarity among ${latent^{R}}_{real}$, ${latent^{R}}_{fog}$,
and ${latent^{R}}_{night}$. This is achieved by the use of Barlow Twins
objective ($\mathsf{L}_{\mathcal{BT}}$). It attempts to make the cross-
correlation matrix($\mathcal{C}\in\thinspace\mathbb{R}^{D\times D}$) computed
from twin embeddings in the same procedure of [75], as close to the identity
matrix as possible. More details about calculating the cross-correlation
matrix can be found in Appendix C. The $\mathsf{L}_{\mathcal{BT}}$ enforces
reduction of redundancy within an embedding and invariance between two
embeddings by estimating the deviation of the cross-correlation
matrix($\mathcal{C}$) from the identity matrix. This deviation is defined as a
loss and is backpropagated to update the network weights. The general Barlow
Twins objective ($\mathsf{L}_{\mathcal{BT}}$) used in this work is given as

$\displaystyle\mathsf{L}_{\mathcal{BT}}\triangleq\underbrace{\alpha_{1}\sum\limits_{i}^{D}{{(1-\mathcal{C}_{ii})}^{2}}}_{\text{invariance
term}}+\underbrace{\alpha_{2}\lambda\sum\limits_{i}^{D}\sum\limits_{j\neq
i}^{D}{\mathcal{C}_{ij}}^{2}}_{\text{redundancy reduction term}}.$ (6)

where $\alpha_{1}$, $\alpha_{2}$ are our empirical weighting parameters while
$\lambda$ is a standard weight parameter suggested by [75]. The
$\mathcal{C}_{ii}$ represents diagonal elements while $\mathcal{C}_{ij}$
represents off-diagonal elements of the matrix $\mathcal{C}$. Indexing the
matrix is done by using $i$, $j$ along dimension $\mathbb{R}^{D\times D}$.
From (6) it is clear that diagonal elements of the cross-correlation matrix
are forced to be closer to 1 while the off-diagonal elements of the cross-
correlation matrix are forced to be 0. To impose domain invariance on all
latent embeddings, four cross-correlation matrices
$\mathcal{C}\in\thinspace\mathbb{R}^{D\times D}$, e.g., two for translation
latent pairs and two for rotation latent pairs, are calculated and obtained as
follows

$\displaystyle\mathcal{C}^{T^{fog}}=\mathcal{C}({latent^{T}}_{real},{latent^{T}}_{fog}),$
(7)
$\displaystyle\mathcal{C}^{T^{night}}=\mathcal{C}({latent^{T}}_{real},{latent^{T}}_{night}),$
(8)
$\displaystyle\mathcal{C}^{R^{fog}}=\mathcal{C}({latent^{R}}_{real},{latent^{R}}_{fog}),$
(9)
$\displaystyle\mathcal{C}^{R^{night}}=\mathcal{C}({latent^{R}}_{real},{latent^{R}}_{night}).$
(10)

The Barlow Twins objective [75] in (6) is applied for the four cross-
correlation matrices in (7, 8, 9, 10) and combined as
$\mathsf{L}_{\mathcal{BT}_{total}}$. To further enforce domain invariance,
$\mathsf{L}_{2}$ losses for the combination of the real domain and augmented
domains are calculated. The $\mathsf{L}_{2}$ losses are defined as
$\mathsf{L}_{2_{T_{fog}}}$, $\mathsf{L}_{2_{T_{night}}}$,
$\mathsf{L}_{2_{R_{fog}}}$, and $\mathsf{L}_{2_{R_{night}}}$ and put together
as $\mathsf{L}_{2_{total}}$ and an objective to minimize the error during the
prediction of the final pose is evaluated. This loss, given as
$\mathsf{L}_{\mathcal{P}}$ [24], is calculated for the prediction of
$\widehat{\mathcal{P}_{real}}$, $\widehat{\mathcal{P}_{fog}}$, and
$\widehat{\mathcal{P}_{night}}\thinspace\in\thinspace\mathbb{R}^{7}$ with
respect to the ground truth pose $\mathcal{P}_{GT}$ and combined as
$\mathsf{L}_{\mathcal{P}_{total}}$. The final training objective
$\mathsf{L}_{\mathcal{DA}}$ for network optimization is the combination of the
three objectives $\mathsf{L}_{\mathcal{BT}_{total}}$,
$\mathsf{L}_{2_{total}}$, and $\mathsf{L}_{\mathcal{P}_{total}}$ as following

$\mathsf{L}_{\mathcal{BT}_{total}}=\mathsf{L}_{\mathcal{BT}_{T_{fog}}}+\mathsf{L}_{\mathcal{BT}_{T_{night}}}+\mathsf{L}_{\mathcal{BT}_{R_{fog}}}+\mathsf{L}_{\mathcal{BT}_{R_{night}}}$,

(11)

$\mathsf{L}_{2_{total}}=\mathsf{L}_{2_{T_{fog}}}+\mathsf{L}_{2_{T_{night}}}+\mathsf{L}_{2_{R_{fog}}}+\mathsf{L}_{2_{R_{night}}}$,

(12)
$\displaystyle\mathsf{L}_{\mathcal{P}_{total}}=\mathsf{L}_{\mathcal{P}_{real}}+\mathsf{L}_{\mathcal{P}_{fog}}+\mathsf{L}_{\mathcal{P}_{night}},$
(13)
$\displaystyle\mathsf{L}_{\mathcal{DA}}=\mathsf{L}_{\mathcal{BT}_{total}}+\mathsf{L}_{2_{total}}+\mathsf{L}_{\mathcal{P}_{total}}.$
(14)

The network is trained end-to-end with the loss function
$\mathsf{L}_{\mathcal{DA}}$ given in (14).

Inference: An important aspect of the proposed framework for absolute pose
estimation is that the architecture of each parallel branch uses shared
weights during training. Since the weights are shared, the trained model can
be used as a single-branch model during inference time. Additionally, it can
be used for inference for an image with any data distribution. In Fig. 2(b),
the inference process for pose estimation of a single image is shown. This
property of the proposed framework ensures that no additional complexity
during inference is added and we achieve faster runtime as will be presented
in the next section.

Table 1: Results for outdoors Cambridge Landmark dataset. The last column
represents the overall rank of different methods.

Method | KingsCollege | OldHospital | ShopFacade | StMarysChurch | Average | Rank | Overall Rank
---|---|---|---|---|---|---|---
| T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T | R |
MSPN [5] | 1.73 | 3.65 | 2.55 | 4.05 | 2.92 | 7.49 | 2.67 | 6.18 | 2.47 | 5.34 | 13 | 10 | 12
IR-Baseline [55] | 1.48 | 4.45 | 2.68 | 4.63 | 0.90 | 4.32 | 1.62 | 6.06 | 1.67 | 4.87 | 9 | 8 | 9
PoseNet [25] | 1.92 | 5.40 | 2.31 | 5.38 | 1.46 | 8.08 | 2.65 | 8.48 | 2.09 | 6.84 | 12 | 13 | 13
PoseNet(Learnable) [24] | 0.99 | 1.06 | 2.17 | 2.94 | 1.05 | 3.97 | 1.49 | 3.43 | 1.43 | 2.85 | 6 | 2 | 3
GeoPoseNet [24] | 0.88 | 1.04 | 3.20 | 3.29 | 0.88 | 3.78 | 1.57 | 3.32 | 1.63 | 2.86 | 8 | 3 | 5
LSTM-PN [71] | 0.99 | 3.65 | 1.51 | 4.29 | 1.18 | 7.44 | 1.52 | 6.68 | 1.30 | 5.52 | 3 | 11 | 8
GPoseNet [9] | 1.61 | 2.29 | 2.62 | 3.89 | 1.14 | 5.73 | 2.93 | 6.46 | 2.08 | 4.59 | 11 | 7 | 10
BayesianPN [23] | 1.74 | 4.06 | 2.57 | 5.14 | 1.25 | 7.54 | 2.11 | 8.38 | 1.92 | 6.28 | 10 | 12 | 11
SVS-Pose [40] | 1.06 | 2.81 | 1.50 | 4.03 | 0.63 | 5.73 | 2.11 | 8.11 | 1.33 | 5.17 | 4 | 9 | 6
MapNet [7] | 1.07 | 1.89 | 1.94 | 3.91 | 1.49 | 4.22 | 2.00 | 4.53 | 1.63 | 3.64 | 7 | 6 | 6
IRPNet [56] | 1.18 | 2.19 | 1.87 | 3.38 | 0.72 | 3.47 | 1.87 | 4.94 | 1.42 | 3.45 | 5 | 5 | 4
$\mathcal{DA}$-model (Ours) | 0.74 | 2.81 | 1.88 | 2.97 | 0.84 | 3.71 | 1.31 | 4.29 | 1.19 | 3.44 | 2 | 4 | 2
MS-Transformer(Optimized) [57, 58] | *(0.83) | 1.47 | *(1.81) | 2.39 | *(0.86) | 3.07 | *(1.62) | 3.99 | 0.96(1.28) | 2.73 | 1 | 1 | 1

*Scene-specific optimized translation vectors are unknown; however, for ranking, we used the overall average result from [58]

Table 2: Results for indoor 7Scenes dataset. The last column represents the
overall rank of different methods.

Method | Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs | Average | Rank | Overall Rank
---|---|---|---|---|---|---|---|---|---|---
| T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) |
MSPN [5] | 0.09 | 4.76 | 0.29 | 10.50 | 0.16 | 13.10 | 0.16 | 6.80 | 0.19 | 5.50 | 0.21 | 6.61 | 0.31 | 11.63 | 0.20 | 8.41 | 4 | 6 | 4
IR-Baseline [55] | 0.18 | 10.00 | 0.33 | 12.40 | 0.15 | 14.30 | 0.25 | 10.10 | 0.26 | 9.42 | 0.27 | 11.10 | 0.24 | 14.70 | 0.24 | 11.72 | 10 | 14 | 12
PoseNet [25] | 0.32 | 8.12 | 0.47 | 14.40 | 0.29 | 12.00 | 0.48 | 7.68 | 0.47 | 8.42 | 0.59 | 8.64 | 0.47 | 13.80 | 0.44 | 10.43 | 13 | 13 | 14
PoseNet(Learnable) [24] | 0.14 | 4.50 | 0.27 | 11.80 | 0.18 | 12.10 | 0.20 | 5.77 | 0.25 | 4.82 | 0.24 | 5.52 | 0.37 | 10.60 | 0.24 | 7.87 | 9 | 4 | 7
GeoPoseNet [24] | 0.13 | 4.48 | 0.27 | 11.30 | 0.17 | 13.00 | 0.19 | 5.55 | 0.26 | 4.75 | 0.23 | 5.35 | 0.35 | 12.40 | 0.23 | 8.12 | 6 | 5 | 6
LSTM-PN [71] | 0.24 | 5.77 | 0.34 | 11.90 | 0.21 | 13.70 | 0.30 | 8.08 | 0.33 | 7.00 | 0.37 | 8.83 | 0.40 | 13.70 | 0.31 | 9.86 | 12 | 11 | 10
GPoseNet [9] | 0.20 | 7.11 | 0.38 | 12.30 | 0.21 | 13.80 | 0.28 | 8.83 | 0.37 | 6.94 | 0.35 | 8.15 | 0.37 | 12.50 | 0.31 | 9.95 | 11 | 12 | 10
BayesianPN [23] | 0.37 | 7.24 | 0.43 | 13.70 | 0.31 | 12.00 | 0.48 | 8.04 | 0.61 | 7.08 | 0.58 | 7.54 | 0.48 | 13.10 | 0.47 | 9.81 | 14 | 10 | 12
Hourglass [36] | 0.15 | 6.17 | 0.27 | 10.80 | 0.19 | 11.60 | 0.21 | 8.48 | 0.25 | 7.01 | 0.27 | 10.20 | 0.29 | 12.50 | 0.23 | 9.53 | 7 | 9 | 9
MapNet [7] | 0.08 | 3.25 | 0.27 | 11.70 | 0.18 | 13.30 | 0.17 | 5.15 | 0.22 | 4.02 | 0.23 | 4.93 | 0.30 | 12.10 | 0.21 | 7.79 | 5 | 3 | 3
AtLoc [72] | 0.10 | 4.07 | 0.25 | 11.40 | 0.16 | 11.80 | 0.17 | 5.34 | 0.21 | 4.37 | 0.23 | 5.42 | 0.26 | 10.50 | 0.20 | 7.56 | 3 | 2 | 2
IPRNet [56] | 0.13 | 5.64 | 0.25 | 9.67 | 0.15 | 13.10 | 0.24 | 6.33 | 0.22 | 5.78 | 0.30 | 7.29 | 0.34 | 11.60 | 0.23 | 8.49 | 7 | 7 | 8
$\mathcal{DA}$-model (Ours) | 0.11 | 6.06 | 0.26 | 11.67 | 0.15 | 12.81 | 0.17 | 8.34 | 0.21 | 6.37 | 0.19 | 8.90 | 0.26 | 12.21 | 0.19 | 9.48 | 2 | 8 | 4
MS-Transformer(Optimized) [57, 58] | *(0.11) | 4.66 | *(0.24) | 9.60 | *(0.14) | 12.19 | *(0.17) | 5.66 | *(0.18) | 4.44 | *(0.17) | 5.94 | *(0.26) | 8.45 | 0.15(0.18) | 7.28 | 1 | 1 | 1

*Scene-specific optimized translation vectors are unknown; however, for ranking, we used the overall average result from [58]

$\mathcal{DA}$-model |  |  |  |
---|---|---|---|---
MS-Transformer |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: Camera trajectory visualization [15] on real Cambridge Landmarks and
7Scenes datasets [17, 25]. Each plot shows the camera trajectory, green for
the ground truth and red for the prediction. From left to right, the testing
trajectories are for the scenes KingsCollege-seq-02, StMarysChurch-seq-13,
Office-seq-06, and Heads-seq-01. From the $1^{st}$ row, it is clear that the
$\mathcal{DA}$-model inherently leads to fewer discontinuities and more stable
trajectories than the MS-Transformer[57] ($2^{nd}$ row). More trajectories and
analysis are given in Appendix D.

##  4 Experiments

Datasets: Our approach is evaluated using the Cambridge Landmarks[25] and
7Scenes[17] datasets, which have recently been used as camera pose estimation
benchmark sets. The Cambridge Landmarks dataset includes six scenes $(\sim
900-5500m^{2})$ taken in urban areas. For the comparative analysis, we
considered four scenes (KingsCollege, OldHospital, ShopFace, and
StMarysChurch) from the Cambridge Landmarks dataset that are used for APR
assessments. Captured with a handheld Kinect RGB-D camera, the 7Scenes dataset
contains seven small-scale scenes $(\sim 1-10m^{2})$ depicting indoor
environments. We use all seven scenes from the 7Scenes dataset (Chess, Fire,
Heads, Office, Pumpkin, Kitchen, and Stairs) for a comprehensive performance
assessment. Official train/test splits are used from the datasets. Samples of
dataset images and domain augmentations are shown in Appendix I.

Environment: Following the preprocessing steps performed in [57, 58], the
scene images contained in the dataset are re-scaled so that the dimension of
the shorter side of the image is set to have 256 pixels. The re-scaled image
is then cropped randomly into a square shape (224 $\times$ 224) in order to
train the APR network. Similar to the baselines, it is evaluated using center
crops at test time. The loss function given in (14) is used to optimize the
network by using the Adam optimizer [29] with the learning rate set to
$1e^{-3}$ and weight decay set to $1e^{-4}$. All $\lambda$ in
$\mathsf{L}_{\mathcal{BT}}$ are kept at $5.1e^{-3}$ while $\alpha_{1}$ and
$\alpha_{2}$ are set at $1e^{-7}$ and $1e^{-3}$. More details on the
environment and hyperparameter settings are given in Appendix B.

###  4.1 Comparison with baseline methods

The visual localization results for the Cambridge Landmarks and 7scenes
dataset are presented in Tables 1 and 2, respectively. Median value of
translation error(meters) and rotation error(degrees) are shown for each
target scene. The overall average of median errors calculated across scenes is
also added for comparison. We rank the proposed method and the baseline
methods similarly to [57]. Table 1 shows the results for outdoor scenes. In
Table 1 it is seen that the proposed method achieves the second-best result
for translation prediction while achieving the fourth-best result for rotation
prediction. It should be noted that in the last row of Table 1 we report the
refined (optimized) pose estimation results from [58]. The refinement in [58]
requires additional training for pose encoder and an iterative post-processing
step in addition to the inference performed by the transformer network. On the
other hand, our network works as a simple APR in terms of inference, without
any optimization step (see Section 4.2 for runtime analysis). Even with a
lightweight architecture, the proposed method outperforms computationally
heavy transformer-based architecture without a refinement step [57] in terms
of translation vector estimation for outdoor scenes.

Table 2 shows the results for indoor scenes. In Table 2 it is shown that the
proposed method achieves the second-best result for translation prediction
while achieving the eighth-best result for rotation prediction. Even though
the proposed method achieves consistent results for translation prediction,
for indoor scenes we observed limitations in the rotation prediction. As it is
presented in subsection 4.3, the addition of the MHA component improves
rotation prediction by having access to the features learned by the CNN
backbone. To evaluate how the error in rotation prediction affects the
generation of camera trajectories, we evaluate the camera trajectory for two
outdoor scenes and two indoor scenes using the proposed method and MS-
Transformer. The visualization of camera trajectories is presented in Fig. 3.
It can be observed that, even with a higher predicted error and without any
temporal information, the proposed method leads to less discontinuities and
more stable trajectories.

###  4.2 Computational Complexity Analysis

We perform a computational complexity analysis with the proposed method and
three baseline methods, MS-Transformer, AtLoc, and MapNet. Flop count,
activation count, number of architecture parameters, and memory footprint are
presented, similar to the analysis shown in [47]. Flop count and activation
count are calculated using the tool [50]. The results obtained are presented
in Table 3. It is shown that our method is highly efficient regarding FLOPs,
number of parameters and memory footprint when compared to baseline methods.
The proposed method uses nearly 24 times fewer FLOPs than MS-Transformer and
nearly 13 times fewer than AtLoc. Our model also achieves equivalence to other
methods while having 5-7 times fewer parameters and a memory footprint. In
terms of activations, our model has approximately 12 times fewer activations
when compared to the MS-Transformer while having approximately 16% more
activations when compared to AtLoc and MapNet. More details on various CNN
backbones impact on domain adaptive framework is presented in Appendix G.

###  4.3 Ablation studies

Table 3: Computational complexity analysis in terms of FLOPs, activations,
parameters, and memory. Bold numbers represent the highest efficiency.

Methods |  FLOPs (G) |  Activations (millions) |  Params (millions) |  Memory (MB)
---|---|---|---|---
Ours | 0.237 | 4.4 | 3.847 | 14.2
MS-Transformer [57] | 5.578 | 52.2 | 18.542 | 74.5
AtLoc [72] | 3.068 | 3.8 | 24.448 | 97.9
MapNet [7] | 3.672 | 3.7 | 22.348 | 89.5

Table 4: Impact of individual components $\mathsf{L}_{BT}$, $\mathsf{L}_{2}$,
and MHA

Components | ShopFacade | OldHospital | Chess | Heads |  Memory (MB)
---|---|---|---|---|---
| T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg) |
W/O $\mathsf{L}_{\mathcal{BT}}$ | 0.85 | 4.05 | 2.14 | 3.04 | 0.11 | 6.76 | 0.16 | 13.21 | 14.2
W/O $\mathsf{L}_{2}$ | 0.81 | 4.29 | 2.09 | 2.97 | 0.11 | 6.30 | 0.15 | 13.15 | 14.2
W/O MHA | 0.81 | 4.46 | 1.70 | 3.20 | 0.11 | 6.25 | 0.16 | 13.62 | 14.1
Full | 0.84 | 3.71 | 1.88 | 2.97 | 0.11 | 6.06 | 0.15 | 12.81 | 14.2

Impact of Proposed Components: Table 4 shows how individual components in our
domain adaptive training framework impact the model performance on outdoor
scenes: ShopFacade, OldHospital, and indoor scenes: Chess and Heads. It is
observed that the proposed components $\mathsf{L}_{BT}$, $\mathsf{L}_{2}$, and
MHA impact especially rotation estimation in the scenes analysed.

Table 5: Average of median errors for multiple indoor and outdoor scenes on
the domains used for training: real, foggy, and night. Bold numbers represent
best performance. See more details in Appendix E.

| Average of median errors for different methods
---|---
|  MS-Transformer |  $\mathcal{SB}$-model |  $\mathcal{DA}$-model
Scenes | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg)
KingsCollege | 1.38 | 2.89 | 1.53 | 3.19 | 0.82 | 2.33
OldHospital | 3.29 | 4.09 | 3.00 | 3.84 | 2.15 | 2.87
ShopFacade | 1.11 | 4.43 | 1.45 | 4.87 | 0.78 | 2.83
StMarysChurch | 2.09 | 11.92 | 1.89 | 5.36 | 1.32 | 3.78
Average | 1.97 | 5.83 | 1.97 | 4.31 | 1.27 | 2.95
chess | 0.12 | 6.70 | 0.16 | 7.32 | 0.11 | 6.11
office | 0.21 | 7.80 | 0.24 | 9.03 | 0.17 | 8.27
pumpkin | 0.24 | 5.42 | 0.31 | 6.85 | 0.22 | 6.34
stairs | 0.37 | 9.53 | 0.32 | 11.40 | 0.27 | 12.18
Average | 0.24 | 7.36 | 0.26 | 8.65 | 0.19 | 8.23

MS-Transformer vs proposed $\mathcal{SB}$-model vs proposed
$\mathcal{DA}$-model on domains seen during training: Next, we compare the
proposed $\mathcal{DA}$-model with MS-Transformer and with a single-branch
$\mathcal{SB}$-model trained with the same APR architecture as
$\mathcal{DA}$-model and without the domain adaptive framework on the original
datasets. The results are shown in Table 5. In this comparison, we evaluate
using real domain and domain augmentation corresponding to foggy and night
images from the test set. The average median errors calculated on the three
domains are reported. It is seen that the $\mathcal{DA}$-model outperforms
other methods, which are not domain adaptive, in 16 out of 20 instances. It is
important to note that MS-Transformer and $\mathcal{SB}$-model do not have
access to foggy and night-view images during the training process.

MS-Transformer vs proposed $\mathcal{SB}$-model vs proposed
$\mathcal{DA}$-model on domains unseen during training: Table 6 shows the
comparison between MS-Transformer, $\mathcal{SB}$-model, and
$\mathcal{DA}$-model in three domains unseen during training. The three
domains, mosaic, udnie, and starry are defined as

$\displaystyle\mathcal{I}_{mosaic}=\mathbb{G}_{mosaic}(\mathcal{I}_{real}),$
(15)
$\displaystyle\mathcal{I}_{udnie}=\mathbb{G}_{udnie}(\mathcal{I}_{real}),$
(16)
$\displaystyle\mathcal{I}_{starry}=\mathbb{G}_{starry}(\mathcal{I}_{real}).$
(17)

where $\mathcal{I}_{mosaic}$, $\mathcal{I}_{udnie}$, and
$\mathcal{I}_{starry}$ are the unseen domain images while
$\mathbb{G}_{mosaic}$, $\mathbb{G}_{udnie}$, and $\mathbb{G}_{starry}$
represent the GAN checkpoints adopted from [38] for image translation. It is
seen in Table 6 that the $\mathcal{DA}$-model consistently outperforms the
$\mathcal{SB}$-model for domains unseen during training. When compared to the
MS-Transformer, the $\mathcal{DA}$-model achieves improved performance for 17
of 20 instances. We also analyzed if the learned embeddings by the proposed
method are robust to domains seen and unseen during training. For that, we
qualitatively analyze by plotting embeddings, using dimensionality reduction
techniques. The results in Fig. 4 show that the $\mathcal{DA}$-model generates
similar embeddings for domains seen and unseen during training, leading to
better pose estimation in general. On the other hand, the $\mathcal{SB}$-model
generates dissimilar embeddings for domains seen and unseen during training,
as observed in the second row of Fig. 4. We also evaluate AtLoc [72] for the
same task. However, only the Stairs scene pretrained model is publicly
available, so we provide a detailed evaluation only for that specific scene
(see Table 7).

Table 6: Average of median errors for multiple indoor and outdoor scenes on
domains unused for training: mosaic, udnie and starry. Bold numbers represent
best performance. See more details in Appendix F.

| Average of median errors for different methods
---|---
|  MS-Transformer |  $\mathcal{SB}$-model |  $\mathcal{DA}$-model
Scenes | T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg)
KingsCollege | 2.93 | 4.97 | 7.66 | 9.38 | 2.49 | 4.32
OldHospital | 4.45 | 6.21 | 4.56 | 5.19 | 4.29 | 4.53
ShopFacade | 1.56 | 7.66 | 3.39 | 11.81 | 2.99 | 9.08
StMarysChurch | 12.02 | 67.01 | 15.14 | 19.37 | 6.39 | 9.47
Average | 5.24 | 21.46 | 7.69 | 11.44 | 4.04 | 6.85
chess | 0.20 | 12.88 | 0.34 | 11.15 | 0.20 | 7.96
office | 0.37 | 20.46 | 0.48 | 18.74 | 0.33 | 12.51
pumpkin | 0.48 | 19.03 | 0.55 | 26.94 | 0.41 | 11.74
stairs | 0.44 | 13.49 | 0.51 | 17.72 | 0.37 | 15.76
Average | 0.37 | 16.47 | 0.47 | 18.64 | 0.33 | 11.99

Table 7: Comparison with AtLoc [72]. Average of median errors for unseen
domains with Stairs scene. Bold numbers represent best performance.

Stairs scene |  AtLoc |  $\mathcal{SB}$-model |  $\mathcal{DA}$-model
---|---|---|---
| T(m) | R(deg) | T(m) | R(deg) | T(m) | R(deg)
mosaic | 0.71 | 17.69 | 0.55 | 20.70 | 0.43 | 17.16
udnie | 0.31 | 15.33 | 0.34 | 16.31 | 0.31 | 15.26
starry | 0.75 | 14.81 | 0.64 | 16.14 | 0.38 | 14.87
Average | 0.59 | 15.94 | 0.51 | 17.72 | 0.37 | 15.76

$\mathcal{DA}$-model |
---|---
$\mathcal{SB}$-model |

(a)

(b)

Figure 4: Visualization of the embeddings produced by $\mathcal{SB}$-model and
$\mathcal{DA}$-model for the test split of StMarysChurch scene in 2 dimensions
by using t-SNE [67, 22] plots. Each subplot represents the latent embedding
space of different domains. From the $1^{st}$ row, it can be seen that
$\mathcal{DA}$-model produces similar embeddings to seen and unseen domains
while the $2^{nd}$ row, shows that $\mathcal{SB}$-model produces dissimilar
embeddings. From left to right, the domains are real, foggy, night, mosaic,
udnie, and starry.

##  5 Conclusion

In this paper, we propose a domain-adaptive training framework for improving
generalization to unseen distributions during inference. Our model is as light
as a single branch APR during inference. We exploit the Barlow Twins objective
which is introduced for a self-supervised learning framework to enforce domain
invariance and redundancy reduction in learned latent embeddings. Even with a
lightweight architecture, the $\mathcal{DA}$-model ranks 2nd with the
Cambridge landmarks dataset and ranks 4th with the 7Scenes dataset. Also, our
method outperforms the state-of-the-art MS-Transformer with unseen domains. By
analysis, we prove that our model is lightweight in terms of the number of
parameters, memory, FLOPs and activations. Our domain adaptive framework is
evaluated with a MobileNetV3-Large based APR. However, the proposed training
framework is general and can be used to retrain any available APRs to improve
their generalization to unseen domains.

## References

  * [1] Mary B Alatise and Gerhard P Hancke.  Pose estimation of a mobile robot based on fusion of imu data and vision data using an extended kalman filter.  Sensors, 17(10):2164, 2017.
  * [2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5297–5307, 2016.
  * [3] R Avenash and P Viswanath.  Semantic segmentation of satellite images using a modified cnn with hard-swish activation function.  In VISIGRAPP (4: VISAPP), pages 413–420, 2019.
  * [4] Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool.  Speeded-up robust features (surf).  Computer vision and image understanding, 110(3):346–359, 2008.
  * [5] Hunter Blanton, Connor Greenwell, Scott Workman, and Nathan Jacobs.  Extending absolute pose regression to multiple scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 38–39, 2020.
  * [6] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6684–6692, 2017.
  * [7] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz.  Geometry-aware learning of maps for camera localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616–2625, 2018.
  * [8] Guillaume Bresson, Zayed Alsayed, Li Yu, and Sébastien Glaser.  Simultaneous localization and mapping: A survey of current trends in autonomous driving.  IEEE Transactions on Intelligent Vehicles, 2(3):194–220, 2017.
  * [9] Ming Cai, Chunhua Shen, and Ian Reid.  A hybrid probabilistic model for camera relocalization.  BMVC Press, 2019.
  * [10] Dimitris Chatzopoulos, Carlos Bermejo, Zhanpeng Huang, and Pan Hui.  Mobile augmented reality survey: From where we are to where we go.  Ieee Access, 5:6917–6950, 2017.
  * [11] Kefan Chen, Noah Snavely, and Ameesh Makadia.  Wide-baseline relative camera pose estimation with directional learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3258–3268, 2021.
  * [12] Ronald Clark, Sen Wang, Andrew Markham, Niki Trigoni, and Hongkai Wen.  Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6856–6864, 2017.
  * [13] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 8092–8101, 2019.
  * [14] Sovann En, Alexis Lechervy, and Frédéric Jurie.  Rpnet: An end-to-end network for relative camera pose estimation.  In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  * [15] Alexander Fabisch.  pytransform3d: 3d transformations for python.  Journal of Open Source Software, 4(33):1159, 2019.
  * [16] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [17] Ben Glocker, Shahram Izadi, Jamie Shotton, and Antonio Criminisi.  Real-time rgb-d camera relocalization.  In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 173–179. IEEE, 2013.
  * [18] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [19] Richard I Hartley.  In defense of the eight-point algorithm.  IEEE Transactions on pattern analysis and machine intelligence, 19(6):580–593, 1997.
  * [20] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [21] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [22] Plotly Technologies Inc.  Collaborative data science, 2015.
  * [23] Alex Kendall and Roberto Cipolla.  Modelling uncertainty in deep learning for camera relocalization.  In 2016 IEEE international conference on Robotics and Automation (ICRA), pages 4762–4769. IEEE, 2016.
  * [24] Alex Kendall and Roberto Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5974–5983, 2017.
  * [25] Alex Kendall, Matthew Grimes, and Roberto Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE international conference on computer vision, pages 2938–2946, 2015.
  * [26] Sungkwan Kim, Inhwan Kim, Luiz Felipe Vecchietti, and Dongsoo Har.  Pose estimation utilizing a gated recurrent unit network for visual localization.  Applied Sciences, 10(24):8876, 2020.
  * [27] Seungwook Kim, Juhong Min, and Minsu Cho.  Transformatcher: Match-to-match attention for semantic correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8697–8707, 2022.
  * [28] Taeyoung Kim, Luiz Felipe Vecchietti, Kyujin Choi, Sangkeum Lee, and Dongsoo Har.  Machine learning for advanced wireless sensor networks: A review.  IEEE Sensors Journal, 21(11):12379–12397, 2020.
  * [29] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [30] Alex Krizhevsky and Geoff Hinton.  Convolutional deep belief networks on cifar-10.  Unpublished manuscript, 40(7):1–9, 2010.
  * [31] Abhijit Kundu, Yin Li, and James M Rehg.  3d-rcnn: Instance-level 3d object reconstruction via render-and-compare.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3559–3568, 2018.
  * [32] Zakaria Laskar, Iaroslav Melekhov, Surya Kalia, and Juho Kannala.  Camera relocalization by computing pairwise relative poses using convolutional neural network.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 929–938, 2017.
  * [33] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [34] Diogo C Luvizon, Hedi Tabia, and David Picard.  Human pose regression by combining indirect part detection and contextual information.  Computers & Graphics, 85:15–22, 2019.
  * [35] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In Proceedings of the European conference on computer vision (ECCV), pages 116–131, 2018.
  * [36] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu.  Image-based localization using hourglass networks.  In Proceedings of the IEEE international conference on computer vision workshops, pages 879–886, 2017.
  * [37] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu.  Relative camera pose estimation using convolutional neural networks.  In International Conference on Advanced Concepts for Intelligent Vision Systems, pages 675–687. Springer, 2017.
  * [38] Rusty Mina.  fast-neural-style: Fast style transfer in pytorch!  https://github.com/iamRusty/fast-neural-style-pytorch, 2018.
  * [39] Sumit Mishra, Praveen Kumar Rajendran, and Dongsoo Har.  Socially acceptable route planning and trajectory behavior analysis of personal mobility device for mobility management with improved sensing.  In International Conference on Robot Intelligence Technology and Applications, pages 53–65. Springer, 2021.
  * [40] Tayyab Naseer and Wolfram Burgard.  Deep regression for monocular camera-based 6-dof global localization in outdoor environments.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1525–1530. IEEE, 2017.
  * [41] David Nistér.  An efficient solution to the five-point relative pose problem.  IEEE transactions on pattern analysis and machine intelligence, 26(6):756–770, 2004.
  * [42] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  In Proceedings of the IEEE international conference on computer vision, pages 3456–3465, 2017.
  * [43] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [44] James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman.  Object retrieval with large vocabularies and fast spatial matching.  In 2007 IEEE conference on computer vision and pattern recognition, pages 1–8. IEEE, 2007.
  * [45] Fabio Pizzati, Pietro Cerri, and Raoul de Charette.  Comogan: continuous model-guided image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14288–14298, 2021.
  * [46] Fabio Pizzati, Jean-François Lalonde, and Raoul de Charette.  Manifest: Manifold deformation for few-shot image translation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  * [47] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020.
  * [48] Praveen Kumar Rajendran, Sumit Mishra, Luiz Felipe Vecchietti, and Dongsoo Har.  Relmobnet: End-to-end relative camera pose estimation using a robust two-stage training.  International Worshop on Distributed Smart Cameras (IWDSC), European Conference on Computer Vision Workshops, 2022.
  * [49] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  arXiv preprint arXiv:1710.05941, 2017.
  * [50] Facebook Research.  fvcore library - facebook research.  https://github.com/facebookresearch/fvcore/.
  * [51] Chris Rockwell, Justin Johnson, and David F. Fouhey.  The 8-point algorithm as an inductive bias for relative pose prediction by vits.  In 3DV, 2022.
  * [52] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  Orb: An efficient alternative to sift or surf.  In 2011 International conference on computer vision, pages 2564–2571. Ieee, 2011.
  * [53] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12716–12725, 2019.
  * [54] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938–4947, 2020.
  * [55] Torsten Sattler, Qunjie Zhou, Marc Pollefeys, and Laura Leal-Taixe.  Understanding the limitations of cnn-based absolute camera pose regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3302–3312, 2019.
  * [56] Yoli Shavit and Ron Ferens.  Do we really need scene-specific pose encoders?  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3186–3192. IEEE, 2021.
  * [57] Yoli Shavit, Ron Ferens, and Yosi Keller.  Learning multi-scene absolute pose regression with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2733–2742, 2021.
  * [58] Yoli Shavit and Yosi Keller.  Camera pose auto-encoders for improving pose regression.  In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
  * [59] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8922–8931, 2021.
  * [60] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei.  Integral human pose regression.  In Proceedings of the European conference on computer vision (ECCV), pages 529–545, 2018.
  * [61] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and Mohammad Norouzi.  Discovery of latent 3d keypoints via end-to-end geometric reasoning.  Advances in neural information processing systems, 31, 2018.
  * [62] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [63] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7199–7209, 2018.
  * [64] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820–2828, 2019.
  * [65] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [66] Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla.  24/7 place recognition by view synthesis.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1808–1817, 2015.
  * [67] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [69] Luiz Felipe Vecchietti, Taeyoung Kim, Kyujin Choi, Junhee Hong, and Dongsoo Har.  Batch prioritization in multigoal reinforcement learning.  IEEE Access, 8:137449–137461, 2020.
  * [70] Luiz Felipe Vecchietti, Minah Seo, and Dongsoo Har.  Sampling rate decay in hindsight experience replay for robot control.  IEEE Transactions on Cybernetics, 2020.
  * [71] Florian Walch, Caner Hazirbas, Laura Leal-Taixe, Torsten Sattler, Sebastian Hilsenbeck, and Daniel Cremers.  Image-based localization using lstms for structured feature correlation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 627–637, 2017.
  * [72] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham.  Atloc: Attention guided camera localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, number 06 in 34, pages 10393–10401, 2020.
  * [73] Chenhao Yang, Yuyi Liu, and Andreas Zell.  Rcpnet: Deep-learning based relative camera pose estimation for uavs.  In 2020 International Conference on Unmanned Aircraft Systems (ICUAS), pages 1085–1092. IEEE, 2020.
  * [74] Tien-Ju Yang, Andrew Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam.  Netadapt: Platform-aware neural network adaptation for mobile applications.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 285–300, 2018.
  * [75] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [76] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun.  Shufflenet: An extremely efficient convolutional neural network for mobile devices.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018.
  * [77] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  arXiv preprint arXiv:1611.01578, 2016.

Appendix

##  Appendix A MobileNetV3 Architecture

In [20], Howard et al. proposed a MobileNet architecture with a complementary
approach using search procedures such as platform-aware network architecture
search(NAS) [77] and NetAdapt [74]. Combining those techniques and novel
architecture advances, MobileNetV3 demonstrated its efficacy in improving the
state of the art in classification, object detection and dense prediction
tasks i.e., segmentation. The MobileNet model architecture found through the
search procedure was observed to be computationally heavy in earlier layers
and a few of the last layers. Therefore, to enhance efficiency, those specific
layers are modified such that the model maintains accuracy. Furthermore,
layers were upgraded with a novel version of the swish (18)[49] nonlinear
activation functions called h-swish(20) to enable faster computation with
quantization-friendly properties. It also leveraged squeeze and excitation
blocks [21] and replaced the sigmoid function with an efficient hard-
sigmoid(19) [3].

$\displaystyle\operatorname{swish}(x)=x\cdot\operatorname{\sigma}(x)$ (18)
$\displaystyle\operatorname{h-sigmoid}[x]=\frac{\operatorname{ReLU6}(x+3)}{6}$
(19)
$\displaystyle\operatorname{h-swish}[x]=x\frac{\operatorname{ReLU6}(x+3)}{6}$
(20)

where $\operatorname{\sigma}(.)$ represents the conventional sigmoid function
while $\operatorname{ReLU6}$ is adopted from [30]. Please check Tables 1, 2
and Fig. 6 of [20] for layer-wise details, similarity between
$\operatorname{swish}$ and $\operatorname{h-swish}$ and similarity between
sigmoid($\operatorname{\sigma}$) and hard-sigmoid. Also, additional
explanations on the efficient redesign of the original last stage
architectures found by search methods are presented in Fig. 5 of [20]. Two
MobileNetV3 models were investigated in [20]: MobileNetV3-Large and
MobileNetV3-Small. More details regarding the search performed in this paper
to select the final CNN backbone are presented in Appendix G.

##  Appendix B Environment and Hyperparameters

The proposed method is implemented in PyTorch [43] and inspired by [58]. The
models are trained on a Ubuntu 18.04 platform 32GB RAM machine with 2 $\times$
NVIDIA GeForce GTX 1080 Ti GPUs. In the multi-head attention module, the
number of heads is set to be 7 as the embedding dimension(49) needs to be
divisible by the number of heads according to [68, 43]. Attention dropout is
set at 0.0025. We trained all the 7Scenes APRs up to 120 epochs with a
learning rate scheduler [43] with the step-size set to 20, gamma set to 0.5,
and batch size set to 32. Cambridge Landmarks APRs are trained for 600 epochs
with a learning rate scheduler with the step-size set to 150, gamma set to
0.1, and batch size set to 64. The latent embedding dimension $D$ is equal to
256 for Cambridge Landmarks and equal to 512 for 7Scenes.

##  Appendix C Review of details in the algorithm

###  C.1 Barlow Twins objective [75]

The cross-correlation matrix($\mathcal{C}$), for latent embeddings ($z^{A}$
and $z^{B}$), is calculated similarly to [75] as follows

$\displaystyle\mathcal{C}_{ij}\triangleq\frac{\sum_{b}z^{A}_{b,i}z^{B}_{b,j}}{\sqrt{\sum_{b}{(z^{A}_{b,i})}^{2}}\sqrt{\sum_{b}{(z^{B}_{b,j})}^{2}}}$
(21)

where $i,j$ are indexes to the embeddings and $b$ are indexes to the batch
samples in eqn.(21) as explained in [75]. The Barlow Twins objective as
presented in this section and used to train our method is shown in detail in
Algorithm 1.

Algorithm 1 PyTorch pseudo-code for the Barlow Twins objective used to train
the proposed method. The pseudo-code is inspired by [75]

⬇

1# lambda: weight on the off-diagonal terms

2# N: batch size

3# D: dimensionality of the embeddings

4# mm: matrix-matrix multiplication

5# off_diagonal: off-diagonal elements of a matrix

6# eye: identity matrix

7class BarlowTwinsLoss(nn.Module):

8 """

9 A class to represent simple Barlow Twins objective

10 """

11 def __init__(self, lambd=0.0051, eps= 1e-5):

12 super(BarlowTwinsLoss, self).__init__()

13 self.lambd = lambd

14 self.eps = eps # epsilon for numerical stability

15 def off_diagonal(self, x):

16 # return a flattened view of the off-diagonal elements of a square matrix

17 n, m = x.shape

18 assert n == m

19 return x.flatten()[:-1].view(n - 1, n + 1)[:, 1:].flatten()

20 def forward(self, z_a, z_b):

21 """

22 :param z_a: (torch.Tensor) batch of real latents, a NxD tensor

23 :param z_b: (torch.Tensor) batch of augmented domain latents, a NxD tensor

24 :return: invariance term, redundancy reduction term

25 """

26 # normalize repr. along the batch dimension

27 z_a_norm = (z_a - z_a.mean(0)) / (z_a.std(0) + self.eps) # NxD

28 z_b_norm = (z_b - z_b.mean(0)) / (z_b.std(0) + self.eps) # NxD

29

30 # cross-correlation matrix

31 N = z_a.shape[0]

32 c = torch.mm(z_a_norm.T, z_b_norm) / N # DxD

33

34 # loss

35 D = z_a.shape[1] # Embedding dimension

36 c_diff = (c - torch.eye(D).to(’cuda’)).pow(2) # DxD

37

38 # multiply off-diagonal elements of c_diff by lambda

39 redundancy_reduction = self.off_diagonal(c_diff)

40 invariance_term = torch.diagonal(c_diff)

41 return invariance_term.sum(), self.lambd * redundancy_reduction.sum()

###  C.2 Pose loss [24]

The geometric loss function adopted from [24] is used in this work.
$\mathsf{L}_{\mathbf{x}}$ measures the norm of deviation between predicted
translation and ground truth translation while $\mathsf{L}_{\mathbf{q}}$
measures the deviation between ground truth quaternion and unit normalized
quaternion of the prediction.

$\displaystyle\mathsf{L}_{\mathcal{P}}=\mathsf{L}_{\mathbf{x}}\exp(-\hat{s_{\mathbf{x}}})+\hat{s_{\mathbf{x}}}+\mathsf{L}_{\mathbf{q}}\exp(-\hat{s_{\mathbf{q}}})+\hat{s_{\mathbf{q}}}$
(22)

where $\hat{s_{\mathbf{x}}}$ and $\hat{s_{\mathbf{q}}}$ take arbitrary initial
values in eqn.(22) as they are parameters learned during training.

###  C.3 Foggy and Night view images augmentation

ManiFest [46] code and checkpoints are publicly
available111https://github.com/cv-rits/ManiFest. Also, CoMoGAN [45] code and
checkpoints are publicly available 222https://github.com/cv-rits/CoMoGAN. The
clear2fog.pth checkpoint from ManiFest is used to augment all the images in
the original datasets [25, 17] to foggy images. Similarly, in the case of
CoMoGAN, the checkpoint in the official repository is used with the sun’s
angle $\phi$ being set to 2.1 to augment all the images in the original
datasets to night view images.

##  Appendix D Qualitative analysis

|  $\mathcal{DA}$-model(Ours) |  $\mathcal{SB}$-model(Ours) | MS-Transformer [57]
---|---|---|---
Cambridge T(m) |  |  |
Cambridge R(deg) |  |  |
7Scenes T(m) |  |  |
7Scenes R(deg) |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 5: Cumulative histogram of prediction errors in the test splits with
rotation(deg) and translation(m). $1^{st}$, $2^{nd}$, $3^{rd}$ columns, depict
the performance of $\mathcal{DA}$-model, $\mathcal{SB}$-model, and MS-
Transformer [57], respectively. The accuracy of the estimation increases with
the area under the curve.

$\mathcal{SB}$-model |  |  |  |
---|---|---|---|---

(a)

(b)

(c)

(d)

Figure 6: $\mathcal{SB}$-model Camera Trajectory visualization on Cambridge
and 7Scenes dataset [17, 25]. Each plot shows the camera trajectory (green for
the ground truth and red for the prediction). From left to right, the testing
sequences are KingsCollege-seq-02, StMarysChurch-seq-13, Office-seq-06 and
Heads-seq-01.

Chess

$\mathcal{DA}$-model |
---|---
$\mathcal{SB}$-model |

(a)

(b)

Redkitchen

$\mathcal{DA}$-model |
---|---
$\mathcal{SB}$-model |

(c)

(d)

Figure 7: Visualization of the embeddings produced by $\mathcal{SB}$-model and
$\mathcal{DA}$-model for the test split of Chess and Redkitchen scene in 2
dimensions by using t-SNE [67, 22] plots. Each subplot represents the latent
embedding space of different domains. From the $1^{st}$ row, it can be seen
that the $\mathcal{DA}$-model produces similar embeddings to seen and unseen
domains while the $2^{nd}$ row, shows that $\mathcal{SB}$-model produces
dissimilar embeddings. From left to right, the domains are real, foggy, night,
mosaic, udnie, and starry.

In Fig. 5, a histogram with the predictions of $\mathcal{DA}$-model,
$\mathcal{SB}$-model and MS-Transformer [57] in the test splits of Cambridge
landmarks [25] and 7Scenes [17] benchmarks on real distributions is shown. It
can be seen that overall, the $\mathcal{DA}$-model has a better estimate than
the $\mathcal{SB}$-model. However, MS-Transformer has better estimation than
other methods. Trajectories of the $\mathcal{SB}$-model for the same sequences
as that of Fig. 3(main text) are presented in Fig. 6. It can be observed that
even the $\mathcal{SB}$-model can generate stable trajectories in the test
sequences, unlike MS-Transformer. Similar to Fig. 4(main text), the latent
embedding visualizations are provided for additional scenes in Fig. 7. It is
seen that $\mathcal{DA}$-model almost obtains identical embeddings for seen
domains(real, foggy, night) and reasonably closer embeddings to unseen
domains(mosaic, udnie, starry). To unseen domains, however, the SB-model
limitations are apparently visible.

##  Appendix E Results from Scenes seen during Training

Table 8: Median errors for multiple indoor and outdoor scenes on domains used
for training: Real, Foggy, Night. Full Version of Table 5(main text).

Real Foggy Night Average Methods / Scenes T(m) R(deg) T(m) R(deg) T(m) R(deg)
T(m) R(deg) KingsCollege 0.83 1.47 1.27 3.47 2.03 3.72 1.38 2.89 OldHospital
1.81 2.39 4.65 6.02 3.39 3.87 3.29 4.09 ShopFacade 0.86 3.07 1.27 4.43 1.20
5.78 1.11 4.43 StMarysChurch 1.62 3.99 2.39 22.36 2.26 9.39 2.09 11.92 Average
1.28 2.73 2.40 9.07 2.22 5.69 1.97 5.83 Chess 0.11 4.66 0.14 8.42 0.13 7.03
0.12 6.70 Office 0.17 5.66 0.20 7.71 0.26 10.03 0.21 7.80 Pumpkin 0.18 4.44
0.23 5.90 0.31 5.90 0.24 5.42 Stairs 0.26 8.45 0.55 9.83 0.30 10.31 0.37 9.53
MS-Transformer Average 0.18 5.80 0.28 7.97 0.25 8.32 0.24 7.36 KingsCollege
0.84 2.58 1.43 2.77 2.32 4.21 1.53 3.19 OldHospital 1.63 3.20 4.08 3.72 3.29
4.59 3.00 3.84 ShopFacade 0.76 3.98 1.66 4.47 1.94 6.16 1.45 4.87
StMarysChurch 1.28 4.47 1.97 5.67 2.44 5.92 1.89 5.36 Average 1.13 3.56 2.28
4.16 2.50 5.22 1.97 4.31 Chess 0.11 5.34 0.19 8.29 0.17 8.32 0.16 7.32 Office
0.18 7.22 0.23 8.34 0.32 11.52 0.24 9.03 Pumpkin 0.24 5.47 0.35 8.23 0.34 6.84
0.31 6.85 Stairs 0.26 11.25 0.38 11.37 0.32 11.57 0.32 11.40
$\mathcal{SB}$-model(Ours) Average 0.20 7.32 0.29 9.06 0.29 9.56 0.26 8.65
KingsCollege 0.74 2.81 0.85 2.61 0.88 3.07 0.82 2.33 OldHospital 1.88 2.97
2.54 3.29 2.02 3.07 2.15 2.87 ShopFacade 0.84 3.71 0.72 3.52 0.79 3.32 0.78
2.83 StMarysChurch 1.31 4.29 1.36 5.05 1.28 4.48 1.32 3.78 Average 1.19 3.45
1.37 3.62 1.24 3.49 1.27 2.95 Chess 0.11 6.06 0.11 5.90 0.11 6.37 0.11 6.11
Office 0.17 8.34 0.17 8.17 0.18 8.30 0.17 8.27 Pumpkin 0.21 6.37 0.22 6.41
0.23 6.24 0.22 6.34 Stairs 0.26 12.21 0.28 11.97 0.26 12.37 0.27 12.18
$\mathcal{DA}$-model(Ours) Average 0.19 8.25 0.19 8.11 0.20 8.32 0.19 8.23

As given in C.3, the original datasets are augmented to foggy and night view
images. Collectively, the original dataset i.e, (real domain) and augmented
domains(foggy, night) take part in the domain adaptive training framework as
explained in the main text. As per one of the goals of the domain adaptive
training framework, the $\mathcal{DA}$-model is expected to produce similar
prediction performance for all these domains. Table 8 shows the full results
for Table 5(main text) as presented in the manuscript. It presents the scene-
wise and domain-wise predictions for both datasets along with averages. In the
detailed results, the $\mathcal{DA}$-model produces similar predictions across
all three domains whereas the $\mathcal{SB}$-model and the MS-Transformer
prediction results seem to be dissimilar. Also, variance in prediction for
different domains is higher in MS-Transformer when compared to
$\mathcal{SB}$-model.

##  Appendix F Results from Scenes not seen during Training

Table 9: Median errors for multiple indoor and outdoor scenes on domains not
used for training: Mosaic, Starry, and Udnie. Full Version of Table 6(main
text).

Mosaic-style Udnie-style Starry-style Average Methods / Scenes T(m) R(deg)
T(m) R(deg) T(m) R(deg) T(m) R(deg) KingsCollege 3.28 4.44 1.95 3.18 3.57 7.28
2.93 4.97 OldHospital 5.65 6.02 2.35 3.80 5.35 8.82 4.45 6.21 ShopFacade 2.11
7.49 1.13 3.66 1.45 11.82 1.56 7.66 StMarysChurch 10.52 73.65 7.34 29.84 18.19
97.53 12.02 67.01 Average 5.39 22.90 3.19 10.12 7.14 31.36 5.24 21.46 Chess
0.28 17.80 0.15 9.00 0.18 11.86 0.20 12.88 Office 0.55 31.44 0.26 14.21 0.30
15.73 0.37 20.46 Pumpkin 0.52 20.16 0.48 19.90 0.44 17.03 0.48 19.03 Stairs
0.60 16.54 0.37 13.08 0.36 10.85 0.44 13.49 MS-Transformer Average 0.49 21.49
0.32 14.05 0.32 13.87 0.37 16.47 KingsCollege 9.55 8.70 3.89 5.58 9.55 13.87
7.66 9.38 OldHospital 5.56 6.29 3.74 4.75 4.37 4.53 4.56 5.19 ShopFacade 4.90
14.44 2.42 10.68 2.84 10.30 3.39 11.81 StMarysChurch 19.78 24.11 5.17 9.31
20.46 24.70 15.14 19.37 Average 9.95 13.38 3.81 7.58 9.30 13.35 7.69 11.44
Chess 0.44 12.87 0.24 8.42 0.33 12.15 0.34 11.15 Office 0.65 29.18 0.39 13.24
0.41 13.80 0.48 18.74 Pumpkin 0.65 32.62 0.50 25.98 0.49 22.24 0.55 26.94
Stairs 0.55 20.70 0.34 16.31 0.64 16.14 0.51 17.72 $\mathcal{SB}$-model(Ours)
Average 0.57 23.84 0.36 15.99 0.47 16.08 0.47 18.64 KingsCollege 2.89 4.64
2.40 3.74 2.18 4.57 2.49 4.32 OldHospital 5.07 5.48 4.19 3.75 3.63 4.36 4.29
4.53 ShopFacade 4.31 10.29 2.28 8.49 2.38 8.47 2.99 9.08 StMarysChurch 10.91
12.21 3.89 7.91 4.35 8.29 6.39 9.47 Average 5.80 8.15 3.19 5.97 3.13 6.42 4.04
6.85 Chess 0.28 9.89 0.16 6.87 0.17 7.12 0.20 7.96 Office 0.48 16.97 0.26
10.51 0.27 10.05 0.33 12.51 Pumpkin 0.49 15.23 0.38 9.96 0.37 10.02 0.41 11.74
Stairs 0.43 17.16 0.31 15.26 0.38 14.87 0.37 15.76 $\mathcal{DA}$-model(Ours)
Average 0.42 14.81 0.28 10.65 0.29 10.51 0.33 11.99

Unseen domain augmentations are generated using the methodology and
checkpoints presented in [38]. To obtain mosaic, udnie and starry domains the
following checkpoints are used333https://github.com/rrmina/fast-neural-style-
pytorch: mosaic.pth, udnie.pth, starry.pth. As another advantage of the domain
adaptive framework, it is expected that the $\mathcal{DA}$-model should also
work reasonably to unseen domains. Table 9 shows the full results for Table
6(main text) as presented in the manuscript. Similar to the previous section,
it can be observed that the $\mathcal{DA}$-model outperforms other baseline
methods when unseen domains are evaluated.

##  Appendix G CNN Backbones

Table 10: Various lightweight CNN backbones impact on the proposed domain
adaptive framework’s performance. MobileNetV3-Large demonstrates the better
trade-off over other lightweight CNN backbones considered.

Backbones | ShopFacade | Chess | FLOPs (G) |  Activation (millions) |  Params (millions) | Memory (MB)
---|---|---|---|---|---|---
T(m) | R(deg) | T(m) | R(deg)
MobileNetV3-Large [20] | 0.84 | 3.71 | 0.11 | 6.06 | 0.237 | 4.4 | 3.847 | 14.2
MobileNetV3-Small [20] | 1.01 | 4.56 | 0.12 | 6.63 | 0.062 | 1.4 | 1.246 | 5.1
ShuffleNet V2 0.5× [35] | 0.92 | 5.32 | 0.11 | 6.85 | 0.045 | 1 | 0.89 | 3.7
ShuffleNet V2 1.0× [35] | 0.87 | 5.63 | 0.11 | 7.00 | 0.154 | 1.9 | 1.802 | 7.4
ShuffleNet V2 1.5× [35] | 0.75 | 5.48 | 0.13 | 8.10 | 0.309 | 2.8 | 3.027 | 12.3
MnasNet 0.5x [64] | 1.80 | 8.13 | 0.13 | 7.41 | 0.119 | 3.1 | 1.617 | 6.7
MnasNet 1.0x [64] | 8.68 | 6.51 | 0.11 | 6.64 | 0.341 | 5.5 | 3.782 | 15.4
EfficientNet-B0 [65] | 0.83 | 4.47 | 0.10 | 6.52 | 0.421 | 6.7 | 5.968 | 24.2

###  G.1 Investigated lightweight backbones

Analyzed CNN backbones are the following: MobileNetV3-Large [20],
MobileNetV3-Small [20], ShuffleNet V2 0.5$\times$ [35], ShuffleNet V2
1.0$\times$ [35], ShuffleNet V2 1.5$\times$ [35], MnasNet 0.5$\times$ [64],
MnasNet 1.0$\times$ [64], and EfficientNet-B0 [65]. Here, ShuffleNet V2
represents the next generation of ShuffleNet [76] that exploited pointwise
group convolution and channel shuffle for efficient operations. The MnasNet
considered a new approach, mobile neural architecture search(MNAS), to find a
low latency model. The family of EfficientNets are enabled by the compound
scaling method investigated by Tan et al [65].

###  G.2 Impact of CNN backbones on $\mathcal{DA}$-model

A key contribution of our work is to maintain the lightweight of the neural
network for the pose regression task. With that in consideration, multiple
lightweight CNN backbone’s impacts are studied with the proposed domain
adaptive training framework. Table 10 demonstrates the detailed evaluation of
localization performance in ShopFacade outdoor scene, Chess indoor scene and
computational properties for various CNN backbones. We were able to achieve a
better trade-off between model performance and latency requirements with
MobileNetV3-Large, thus it was the architecture chosen for the entirety of our
study.

##  Appendix H Review of Structure and IR based Methods

###  H.1 Structure Based Localization

In structure-based localization methods, an image is processed to identify
important points and corresponding associated features (descriptors) that are
then used for feature matching based on visual similarity. To perform this
2D-2D correspondence, keypoint descriptors like SURF[4], SIFT[33], FAST[44],
and ORB[52] are used. The results of the feature correspondence are used to
acquire an essential matrix followed by rejecting outliers using RANSAC
(Iterative model conditioning approach). To create a 3D understanding of the
scene, epipolar geometric properties are leveraged [18]. The global 3D scene
is obtained by using keypoint correspondence[16, 19, 41]. The final camera
pose is obtained by using a perspective-n-point(PnP) [16] method. Compared to
the methods to be presented in this paper, traditional structure-based
localization methods are computationally heavy, with slow inference time.

To replace the sub-components that are part of the aforementioned structure-
based localization traditional pipeline, recently, deep learning-based
approaches have been proposed. D2Net[13] implements feature correspondence by
separating detection and description. SuperGlue[54] uses the graph
representation of a scene to leverage graph neural networks to perform
detection and description concurrently. Transformer-based methods like
LoFTR[59] and TransforMatcher[27] carry out feature matching as well as
semantic correspondence matching. Finally, methods such as DSAC [6] and DSAC++
[16] propose a differentiable version of the original RANSAC algorithm.

###  H.2 Image Retrieval

Image retrieval (IR) methods search for images on a large database and then
retrieve images similar to the target image for pose estimation. The database,
part of IR methods, consists of multiple images of the scene in which the
camera pose is known. Intuitively, IR methods search for images that have
poses closer to the query image. The similarity metrics in which the search is
performed are usually performed by using features obtained with global
descriptors enable by methods such as DenseVLAD[66], NetVLAD[2].

Poses retrieved from the database can be considered priors in the pose
estimation process. This characteristic is powerful to scale this pipeline to
both small-scale and large-scale scenes. However, it is important to account
that the sample density of poses in the database significantly affects its
accuracy. As an alternative to obtaining the final pose using IR, for example,
the camera pose can be determined by the weighted average sum of poses from
the top-$k$ most similar images retrieved from the database. If the density is
high enough, another alternative is to consider the pose of the top-$1$ image
as the pose of the query image. IR methods need the preparation of a carefully
designed database and, also, the storage and search of these pose-labelled
databases for inference. Methods that utilize IR encodings [53, 63] achieve
state-of-the-art accuracy on large-scale datasets for visual localization.
These methods are also inspired and hold a similar methodology with
hierarchical localization approaches [42, 55].

##  Appendix I Additional Samples of GAN generated images

| Real | Foggy [46] | Night [45] | Mosaic [38] | Udnie [38] | Starry [38]
---|---|---|---|---|---|---
Heads |  |  |  |  |  |
Chess |  |  |  |  |  |
ShopFacade |  |  |  |  |  |
KingsCollege |  |  |  |  |  |

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

(q)

(r)

(s)

(t)

(u)

(v)

(w)

(x)

Figure 8: Samples of GAN generated images. (a) Real image (b) Foggy image (c)
Night image (d) Mosaic-style image (e) Udnie-style image (f) Starry-style
image

Figure 8 shows the samples for augmented images in different scenes. It can be
seen that the augmentations that take part in the domain adaptive training
framework are relatively weak augmentations(foggy, night) while the unseen
domains used for evaluations are strong augmentations, i.e. augmentations with
a significant deviation from the image’s real distribution.