Miu Lun Lau, Min Long, Jeff Terry 11institutetext: Boise State University,
Boise ID USA,
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext:
Illinois Institute of Technology, Chicago, Illinois,
22email<EMAIL_ADDRESS>

# Automated Materials Spectroscopy Analysis using Genetic Algorithms

Miu Lun Lau 11 Min Long 11 Jeff Terry 22

###### Abstract

We introduce a Genetic Algorithm (GA) based, open-source project to solve
multi-objective optimization problems of materials characterization data
analysis including EXAFS, XPS and nanoindentation. The modular design and
multiple crossover and mutation options make the software extensible for
additional materials characterization applications too. This automation of the
analysis is crucial in the era when instrumentation acquires data orders of
magnitude more rapidly than it can be analyzed by hand. Our results
demonstrated good fitness scores with minimal human intervention.

###### keywords:

Genetic Algorithm, EXAFS, X-Ray

##  1 Introduction

Advanced data analysis techniques for materials characterization data sets are
of growing interest to the material science community due to its capability in
detecting and potentially predicting the properties of materials. The
performance of this process is limited by two main factors: the significant
inputs from users needed to retrieve important structural parameters and the
quantity of high quality data that is now collected due to advances in
instrumentation. The experiential data collected from modern instruments are
orders of magnitude larger than can be analyzed by trained personnel. For
example, the Fourth Generation Synchrotron Light Sources that are coming
online are expected to produce data at rates 2-3 orders of magnitude greater
than the current collection rates that are as high as 6 GB/s [1]. Advances in
processing are necessary to make in-situ real-time characterization of
materials feasible. Processing large quantities of data can result in
reproducibility problems [2] when inexperience users make errors that then
slowing research productivity, which discourages the quest for research
excellence, and inhibits effective technology transfer and manufacturing
innovation.

In order to address the need to analyze massive datasets both quickly and
accurately through solving multi-objective optimization problems, we have been
developing an open-source code base featuring a Genetic Algorithm (GA) that
can analyze a variety of materials characterization data types with minimal
human input to retrieve important parameters [3]. Although it is still under
active development, it has already demonstrated the ability to automatically
analyze extended X-ray Absorption Fine Structure (EXAFS) data, giving a
reproducible description of the local atomic structure of materials.

The GA-based EXAFS analysis package called EXAFS Neo [4] is written primarily
in Python. It requires the installation of Larch software package [5] and
utilizes a version of FEFF8.5l [6] code contained within Larch for calculating
the initial scattering paths. The code was not parallelized and we found that
parallelization was really not needed for effective use as an analysis tool.
However, we strongly recommended that users execute multiple calculations of
the same data set simultaneously for further error analysis. The included
graphical user interface (GUI) allows for populating multiple parameter sets
for simultaneous exploration of parameter space. Several tutorial video
demonstrations of the EXAFS Neo package are available for viewing at the
package download website [4]. Due to the modular design, this open-source
software can be extended to other materials characterization datasets. Our
recent efforts include extensions for the analysis of X-ray Photoelectron
Spectroscopy (XPS) and nanoindentation data.

We will briefly introduce the materials characterization analysis in Section
II, with a focus on the group of parameters not commonly seen in other GA
codes. The design and implementation of GA for this problem is given in
Section III. We evaluate the experimental results and give performance
analysis of the code in Section IV. Finally, a summary of the work is given in
Section V.

##  2 Analysis of Materials Characterization Data

We use EXAFS as an example to demonstrate the process of materials
characterization analysis. The principles discussed here are directly
applicable to the anslysis of other characterization tools such as XPS and
nanoindentation. EXAFS is used to study local structure around a specific
element on the atomic scale. It can be applied to a wide range of materials
including liquids, amorphous solids, and crystalline materials. The absorption
spectrum, $\mu(E)$, with energy, $E$, of X-ray photons can be represented as a
combination of the background $\mu_{0}$ absorption and oscillations due to
scattering events $\chi(E)$, i.e., $\mu(E)=\mu_{0}(1+\chi(E))$. This equation
is usually rewritten using the photoelectron wavenumber
$k=[{\frac{2m}{\hbar^{2}}(E-E_{0})}]^{1/2}$, where $m$ is the electron mass,
$\hbar$ is the Plank’s constant, and $E_{0}$ is called the energy of the
absorption edge. This results in the following equation for $\chi(k)$:

$\chi(k)=\frac{\mu(k)-\mu_{0}(k)}{\mu_{0}(k)}.$ (1)

The oscillations $\chi(k)$ result from the interference of scattered
photoelectron waves. Scattering can involve single or multiple scattering
events. In single scattering events, the electron wave scatters from only one
neighboring atom before returning to the source atom. In multiple scattering
events, the wave may travel to multiple nearby atoms through a variety of
paths before returning to the source atom. The trajectories of the
photoelectron waves are described as paths. These parameters describing these
paths become the gene components of the chromosomes in the GA code.

The measured X-ray absorption spectrum (XAS) [7] consists of two distinct
regions: the X-ray Absorption Near Edge Structure (XANES) region and the
Extended X-ray Absorption Fine Structure (EXAFS) region, which begins at the
end of the XANES region and extends beyond the absorption edge until the
oscillations damp out. The high energy endpoint is dependent upon the nature
of the scattering atoms. The EXAFS region is used to study interatomic
distances, coordination numbers, and lattice dynamics, from which one can
often infer the surface chemistry of complex systems.

The interference that leads to the measured EXAFS oscillations is due to the
interactions of the emitted photoelectron wave, ejected from inner core shells
by resonant radiation, with the waves scattered by neighboring atoms. These
scattered electron waves modulate the wavefunction of the original
photoelection. If the interference is destructive, a photon cannot be absorbed
because the electron wave cannot be created. Conversely, if the interference
is constructive, more photons are absorbed. Sayers et al. [8] was first to
invert the measured experimental EXAFS data into quasi-radial distribution
functions using a simple point scattering theory. Extensions to this simple
methodology result in the full EXAFS equation:

$\chi(k)=\sum_{i(paths)}\frac{(S_{0}^{2}N_{i})F_{i}(k)}{kR_{i}^{2}}e^{-2\sigma_{i}^{2}k^{2}}e^{-2R_{i}/\lambda(k)}\sin[2kR_{i}+\phi_{i}(k)+\delta_{c}(k)],$
(2)

where $i$ represents an individual scattering path, $R_{i}$ is half of the
scattering distance, $\phi_{i}(k)$ is phase shift due to scattering, and
$\delta_{c}(k)$ is the phase shift due to the potential of the absorbing atom.
$N_{i}$ is the degeneracy of the path, and $S_{0}^{2}$ is the amplitude
reduction factor that arises from quantum mechanical considerations.
$S_{0}^{2}$ and $N_{i}$ are coupled together to describe the amplitude of each
scattering path. $F_{i}(k)$ is the effective scattering amplitude for the
waves in path and $\sigma^{2}$ is the Debye-Waller factor, which accounts for
the thermal and static disorder.

We can observe from Eqn. (2) that the EXAFS signal is a summation of
sinusoidal waves of varying amplitudes from scattering paths containing the
neighboring atoms. The waves are inherently spherical in nature and are
affected by the type of atoms, temperature, neighboring atoms, as well as
inelastic losses. The scattering events $\chi(k)$ typically decay rapidly with
increased wave number $k$, which leads to lower signal to noise values at
high-k. The $\chi(k)$ function is often weighted by $k^{2}$ or $k^{3}$ to
highlight the contribution from the high-k regions of the spectrum. Analysis
of the spectrum of $k^{2}\chi(k)$ in terms of $k$ is known as K-space
analysis.

Figure 1: EXAFS spectra of copper metal of K edge in K and R-space. The data
was taken from XASLAB https://xaslib.xrayabsorption.org/spectrum/91/.

A Fourier Transformation can be applied to the K-space data which leads to
R-space EXAFS data. This R-space data is not a true radial distribution
function, unfortunately. It can be used to limit the analysis to a specific
range of individual scattering paths. An example of K and R-space EXAFS data
is shown in Figure 1. The transformed spectra is:

$\chi(r)=\frac{i\delta k}{\sqrt{\pi
N_{max}}}\sum_{n=1}^{N_{max}}\chi(k)\Omega(k)k_{n}^{w}\exp(2i\pi n/N_{max})$
(3)

where $\delta k$ is the spacing in the k-space spectrum; $N_{max}$ is the
array size; $\Omega(k)$ is the input window function, used to reduced
oscillation from truncation of the input spectrum; and $k_{n}^{w}$ is the k
weight applied to the data.

##  3 Design and Implementation of GA Analysis Code

Conventional analysis of EXAFS can be very difficult depending on the number
of scattering paths needed to fit the experimental data. Each path is
characterized by a set of four parameters: $[(S_{0}^{2}N_{i})$, $\Delta
E_{0}$, $\sigma^{2}_{i}$, $R_{i}]$. We use $\Delta E_{0}$ to correct energy
mismatch of the absorption edge between experiment and theory. It should be
identical for all scattering paths from the same calculation. Many software
packages have been developed for EXAFS analysis, including Larch [5], Demeter
[9], WinXAS [10]. However these tools require significant knowledge of
condensed matter physics, and can be difficult for novice users to use. They
can also be quite time-consuming to use.

To address some of these issues, we have been developing an automated
Materials Characterization Software package for EXAFS, XPS, and
nanoindentation analysis. The code is GA-based for such a multi-objective
optimization problem. We should point out we are not the first to apply GA to
EXAFS analysis [11]. However a comprehensive study (e.g., crossover and
mutation options) of GA algorithms and their effects on uncovering the
parameters for materials characterization analysis have not been studied. This
is the goal of this paper.

GA is a heuristic optimization method inspired by the Darwinian theory of
evolution [12][13]. At the start of the algorithm, a population consisting of
$n$ temporary solutions (individuals) is generated randomly throughout the
solution domains. Each solution is considered as a chromosome consisting of
the parameters of interest and each parameter represents a gene of the
chromosome. The GA evaluates the fitness of each solution in a population
using a fitness function to determine the evolution of the next generation of
solutions. To improve the accuracy of the final solution and the convergence
rate of the involved iterations, a number of evolutionary inspired operators
(e.g., crossover, mutation) are applied to each solution throughout subsequent
generations. Since GA is also a stochastic process, it requires multiple runs
of identical parameters to evaluate if the resulting solution reaches a global
optimum. Compared to other search methods, GA operates by performing a multi-
dimensional search, and encourages information exchange among the different
solutions. We will briefly describe how various operators affect the
optimization process and our implementation for EXAFS analysis. The literature
of [14] is listed for reference on general principles of GA.

###  3.1 Chromosome Representations of $[S_{0}^{2},\Delta
E_{0},\sigma^{2},\Delta R_{i}]$

The conventional GA cannot be applied directly but has to be customized for
automated materials characterization analysis due to the reasons discussed in
Section II. Unlike the conventional GA, where binary strings are used as
individual chromosome in each individual, our software uses a floating point
representation of a set of parameters as individual components:
$[S_{0}^{2},\Delta E_{0},\sigma^{2},\Delta R_{i}]$. $\Delta R_{i}$ is used
instead of $R_{i}$ because we must find how to modify the theoretical paths to
reproduce the experimental data. $S_{0}^{2}$ other than ($S_{0}^{2}N_{i}$) is
adopted because we assume the degeneracy parameter $N_{i}$ is static and taken
from the ideal atomic structure of the compounds under evaluation. In
addition, all our parameters are constrained to valid physical ranges to
prevent unreasonable values from being introduced, such as non-positive
amplitude or Debye-Waller factors. When the number of “paths” of interest are
determined, e.g., $n_{path}$, each individual in evolutionary population will
consists of multiple (e.g.,$n_{path}$) sets of parameters.

###  3.2 Crossover and Mutation

To allow convergence toward the global optimal solution, the best solutions of
each generation are selected to be parents of the future populations,
mirroring observation seen in nature. The selection process is based upon the
fitness values in the population. We employed a selection approach similar to
rank selection, where each solution was sorted based on their fitness values,
from best performing to the least. A certain percentage of the best performing
solutions are selected for retention in the populations. To ensure
biodiversity and reduce the risk of getting trapped in local extrema, a
percentage of random solutions is generated as well. The main challenge
regarding the march toward the optimal solution involves maintaining a balance
between population diversity and selective pressure. A large population
diversity leads to slow convergence on the optimum solution, while a low
population diversity leads to premature convergence.

Crossover or Recombination is described as the operation of combining parental
materials of two or more solutions during which the information is inherited
and exchanged to produce a new solution. In nature most species have two
parents but in GA the crossover operations can extend to more than two
parents. Following the crossover operator, two individuals are selected to
generate subsequent individuals throughout the GA. There are numerous
techniques for the crossover operator in the literature.[14] For example, in
the single-point cross over, the chromosomes of two parent solutions are
swapped before and after a single point. In the double-point crossover,
however, there are two cross over points and the chromosomes between the
points are swapped only. We developed three crossover methods in our GA code
and tested them for EXAFS analysis. The details of results will be discussed
in Section IV.

Mutation operators modify existing solution by disturbing them by random
chance. The mutation usually occurs in the “gene” level. For example, an
offspring in binary representation of “0101” can become “0111” depending upon
the probability of mutation or mutation rate. The exploration of additional
portions of phase space by the mutation operator can help to find the global
optimal solution by allowing the escape from local optimal solutions. The
mutation rate is usually set to be very low, but allows for perturbations in
the ranges of values. Conventionally, the mutation operator remains fixed
throughout the entire optimization process. This can lead to ill-condition
where the mutation operator is not sufficient to steer the solution out of a
local extreme, which leads to premature convergence of the solution set. A
self-adaptive mutation operator can mitigate premature convergence by
increasing or decreasing the chance of mutation at each generation, which
allows the algorithm to continuously refine the search area.

In our GA code, we implemented an algorithm based on the Rechenberg 1/5
success rule [15], which increases the mutation probability ($\sigma$) based
on the “success ratio,” $S_{i}$, at the current generation. It is defined as
the probability of generating an improved fitness value compared to the
previous population. This probability can be further extended to the Crossover
operator as well but was limited to the Mutation operator in this work. The
mutation rate will increase in subsequent generation if $S_{i}$ is greater
than 1/5, and decrease if it is less.

###  3.3 Fitness Calculation and Exit Conditions

The fitness calculation is used to compare the quality of individuals within
the given population. Typically, the fitness is computed every generation to
ensure the solution is converging toward the optimal. However, the calculation
can be very expensive and most approaches aim to minimize the number of
fitness function calls. The fitness function employed for EXAFS is determined
by computing the differences between the GA model and experimental data at
each data point using a $\chi^{2}$ model:

$\chi^{2}=\frac{N_{indep}}{N}\sum_{i=1}^{N}\frac{\left(y^{\rm{model}}_{i}-y^{\rm{data}}_{i}\right)^{2}}{\epsilon_{i}^{2}}.$
(4)

$y_{i}^{\rm{model}}$ represents the model data (e.g., $\chi(k)$) at $i$,
$y_{i}^{\rm{data}}$ represents the experimental data, $N_{indep}$ is the
number of independent points, $N$ is the total number of data points, and
$\epsilon_{i}$ is the measure of uncertainty at each data point. Depending
upon the collection methodology of the EXAFS data, the ratio $N_{indep}/N$ can
range from $1/10$ when the EXAFS signal is over-sampled to speed data
collection to 1 when the data is collected stepwise. In fact, this ratio can
vary across the energy range of the collected EXAFS data. For other materials
characterization analysis techniques like XPS and nanoindentation, we can
utilize similar fitness functions (e.g., Eqn. 4) but with the appropriate
theoretical functions used to replace Eqn. 2 to construct the GA model. This
modularized approach makes our GA-based code an excellent platform for
expansion to other applications.

The principles of GA present the reasoning that an optimal solution can be
reached after enough iterations. However, practical applications may deviate
from this ideal case because it is possible that the optimal solution has not
been found before iterations stop. Therefore, it is important to implement
proper exit conditions to terminate the running of the code. Our code uses two
method for determining when the algorithm reaches exit conditions. The first
one is to set a maximum number of iteration by users. The second method is
more adaptive. When the solution doesn’t improves for a number of generations,
we exit with the assumption that an optimal has been reached. In our software,
we provided both methods of controlling the exit condition of the code.

###  3.4 Error Analysis

GA is a probabilistic optimization method where the solution can not be
expected to be identical over repeated runs. It is critical to perform a set
of individual, independent runs (e.g., 50) with varying conditions (e.g.,
population size, generation number, mutation rate) to evaluate the accuracy of
the solution and determine the range of errors. In our GA code, we use the
Global Random Analysis method [16] to estimate errors of the GA-optimized
parameters. The error of each individual parameter (e.g., $S_{0}^{2},\Delta
E_{0},\sigma^{2},\Delta R_{i}$) in the simplest of X-ray absorption cases can
be quantified using a Poisson distribution and the standard deviation is
roughly equal to the square root of the absorption duration time [17]. It is
worth noting that this quantification can only be used when the sample is
uniform in composition and thickness, and cannot be applied to heterogeneous
samples or samples which have suffered any irradiation damage. However, these
limitation do potentially affect the error analysis of this methodology as
they do in conventional EXAFS analysis.

During each generation, the individuals of the best fitness value are stored
and used to construct the covariance matrix which contains the error for each
parameter in the solution. Random perturbations of three parameters are
selected in a bounded range: population size (100-5000), total number of
generations (10-50), and mutation rate (0%-100%). For example, one individual
run of 50 runs could randomly select a condition in the bounded ranges like
population size 200, generation number 30, mutation rate 20% and other runs
may select other conditions. The GA code is run using each set of new starting
conditions. The combined results are utilized for global random analysis.

##  4 Experiment Results

The majority of the code is implemented and written in Python. All our
experiments were conducted on the Idaho National Laboratory High Performance
Computing (HPC) cluster ‘Sawtooth’ which is comprised of Xeon Platinum 8268
Processors. Each optimization was repeated 100 times to compute the standard
deviations and error matrix, and for each experiment we requested four CPU
cores and 12.0 GB of memory. To validate our GA code before applying it to
analysis of real experimental EXAFS data, we first applied the code to a set
of synthetic data, which is generated from five known scattering paths.
Gaussian noise with a signal to noise ratio of 20 was added to the synthetic
spectra.

Table 1: GA fitted parameters and errors from the synthetic data set. Path | $N$ | $S_{0}^{2}$ |  $\Delta$E0 (eV) |  $\sigma^{2}$ (Å2) |  $\Delta$R (Å)
---|---|---|---|---|---
# |  | Model | True | Model | True | Model | True | Model | True
1 | 12 |  0.62$\pm$0.03 | 0.62 |  -0.3$\pm$0.7 | -0.91 |  0.0041$\pm$0.0004 | 0.004 |  0.0512$\pm$0.003 | 0.05
2 | 6 |  0.72$\pm$0.08 | 0.66 |  -0.3$\pm$0.7 | -0.91 |  0.0015$\pm$0.0007 | 0.001 |  0.0118$\pm$0.005 | 0.01
3 | 48 |  0.5$\pm$0.2 | 0.74 |  -0.3$\pm$0.7 | -0.91 |  0.0102$\pm$0.0044 | 0.014 |  0.0615$\pm$0.032 | 0.08
4 | 48 |  0.3$\pm$0.2 | 0.45 |  -0.3$\pm$0.7 | -0.91 |  0.0068$\pm$0.0043 | 0.009 |  0.0467$\pm$0.043 | 0.00
5 | 24 |  0.22$\pm$0.08 | 0.14 |  -0.3$\pm$0.7 | -0.91 |  0.0081$\pm$0.0024 | 0.005 |  0.0538$\pm$0.012 | 0.05

Tab. 1 shows a set of GA fitted parameters with the corresponding errors
generated from the synthetic data. The true values are also listed for
comparison. Since the scattering with single path has the shortest traveling
distance, it has the largest contribution to the absorption spectrum and is of
particular interest. It can be seen from the first row (Path 1), all of the
fitted parameters: $S_{0}^{2}$, $E_{0}$, $\sigma^{2}$, and $\delta_{R}$ match
extremely well to the true values, indicating a high confidence of accuracy
using our GA model. For the weaker scattering paths, the errors are found to
be larger but are comparable to conventional analysis methods.

Figure 2: EXAFS spectra fits of the synthetic experimental spectra with added
Gaussian noise. Left: Spectra were fitted in K-space over the region from 2.5
to 12.5 $\AA^{-1}$. Right: Spectra in R space in a range from 0 to 5 $\AA$.

Fig. 2 shows the GA fitted spectra in both K-space and R-space in comparison
with the true synthetic data. The included Gaussian noise was higher than what
is usually observed in real experimental EXAFS data. This choice was made to
test the performance of our GA code under extreme conditions. It can be seen
that the GA code was able to obtain good matches to the actual values,
especially in lower $k$ ranger, which contribute to the fitted parameters with
the highest level of signal to noise ratio.

Figure 3: The effect of the crossover (red) and mutation (blue) operators on
the overall calculation of fitness value.

Fig. 3 shows the effects of varying both the mutation and crossover operators,
to explore their contributions to the change of the fitness value. We can
observe the average fitness value significantly decreases (i.e., betters)
along with the evolution of generations. The crossover operator has the
largest contribution to the improvement of the fitness value in the early
stages of the evolving generations, which can be seen from the significant
heights of red bars in Fig. 3. The mutation operator contributes less in
general compared to the crossover operator in early stages and even has zero
contribution in a few cases. However, the mutation operator contribution
dominates at the late stages of the evolution. This is because the mutation
probability increases as the generations evolve under the Rechenberg
algorithm, which leads to an increase in amount of phase space probed by the
mutation that occur. Mutation is also used to check if the solution is in
global optimal.

Next, we will employ three metrics to evaluate the quality of our fits in both
K and R spaces: Coefficient of Variation (R2), Mean Absolute Error (MAE), and
Root Mean Square Error (RMSE) [18]. R2 measures the residuals of spectra, MAE
measures errors within the same unit spectrum but gives similar weights to
outliers, and RSME is used to address amplitude errors from outliers.

###  4.1 Crossover Analysis and Mutation Analysis

We provided three crossover methods in our GA code. Users can examine their
effects and select the best option for their specific problems. The first
method “uniform random crossover” utilizes uniform crossover where each gene
is selected randomly from either parent with equal probability. The second
method “AND crossover” utilizes a mixing rule where each gene in child is a
result of an AND logic operation on to each gene from its parents. The third
method“OR crossover” utilizes the similar mixing rule using an OR logic
operator.

Table 2: Error analysis for three crossover and three mutations methods. | Crossover | Mutation
---|---|---
Method | 1(uniform random) | 2(AND) | 3(OR) | 1(maximum) | 2(nested) | 3(metropolis)
R2-K | 0.99856 | 0.95530 | 0.94321 | 0.99900 | 0.98625 | 0.99920
R2-R | 0.99661 | 0.98541 | 0.97431 | 0.99632 | 0.99163 | 0.99656
MAE-K | 0.03912 | 0.17298 | 0.03345 | 0.03389 | 0.11349 | 0.03047
MAE-R | 0.03455 | 0.04561 | 0.04733 | 0.03700 | 0.04119 | 0.03611
RMSE-K | 0.00231 | 0.05333 | 0.00255 | 0.00175 | 0.05250 | 0.00142
RMSE-R | 0.00211 | 0.01044 | 0.00355 | 0.00264 | 0.00604 | 0.00247

Tab. 2 shows that the first method returns the highest contribution to the
true value in both K and R spaces. The second and third methods may produce
lower accuracy than the uniform random crossover in our application. This is
because our solution set only contains a limited number of four parameters,
therefore it is ideal to use uniform random crossover to promote information
exchange and diversity as much as possible between individuals. For this
reason, we adopted the uniform random crossover in the following experiments.

We also provided three mutation methods in our GA code for users. They all use
the same starting mutation probability at each generation $\sigma_{i}$ and
compare it with some random number to determine if a new mutation is
necessary. The first method “maximum mutation” generates a complete new
individual in units of the parameters if a random generated number reaches the
mutation probability. It is a simple method and maximizes the number of
possible mutations in the population. It ensures that sufficient genetic
diversity is introduced within the overall population. The second method
“nested mutation” introduces a secondary random number to control the actual
mutation. The new individual can only be generated in units of gene when the
both random numbers are less than the mutation rate at each generation. This
method fits a more traditional GA where the actual mutation rate for each gene
is typical set very low (e.g., 1 to 5%).

Algorithm 1 Mutation method 2: Nested Mutation. Note: Removing lines 4,5,6,8,9
can lead to Mutation method 1: Maximum Mutation.

0: Mutation Rate $\sigma$, Individual $P_{i}$

0: Individual $P_{i}$

1: for $P_{i}$ in Populations ($P$) do

2: Generate a number $x$ in [0..100]

3: if $x<\sigma$ then

4: for number of variables in each scattering paths do

5: Generate a number $y$ in [0..100]

6: if $y<\sigma$ then

7: Generate new individual

8: end if

9: end for

10: end if

11: end for

Algorithm 2 Mutation method 3: Metropolis Mutation

0: Mutation Rate $\sigma$, Individual $P_{i}$, Fitness $f_{i}^{orig}$

0: Individual $P_{i}$

1: for $P_{i}$ in Populations ($P$) do

2: Generate a random $x$ in [0..100]

3: if $x<\sigma$ then

4: Mutate the Scattering Paths

5: Calculate New Fitness $f_{i}^{mut}$

6: if $f_{i}^{mut}<f_{i}^{orig}$ then

7: Accept the Mutation

8: else if $\exp(-(f_{\mathrm{mut}}-f_{\mathrm{orig}})/K(i))<t$ then

9: Accept the Mutation

10: else

11: Reject the Mutation

12: end if

13: end if

14: end for

The third method “metropolis mutation” is modeled after the Metropolis-Hasting
algorithm [19], which rejects the mutation if the fitness value is less than
current fitness value, or accepts it if the following criterion is satisfied:
$\exp(-(f_{i}^{\mathrm{mut}}-f_{i}^{\mathrm{orig}})/K(i))<t$. The
$f_{i}^{\mathrm{mut}}$ and $f_{i}^{\mathrm{orig}}$ are fitness values after
and before the mutation, $t$ is a random uniform value $\in[0,1]$, and where
$K_{i}K(i)=-\delta_{f}/\ln(1-i/i_{max})$ is a parameter called the cooling
rate. The $\delta_{f}$ is the absolute difference in fitness value between
subsequent generations, $i$ is the current generation number, and $i_{max}$ is
the maximum generation. This cooling rate allows the possibility of accepting
parameters outside of the predefined range, while assuming that the mutation
rate will decrease at a linear rate.

Table 2 compares the different mutation methods. We can see that the value of
R2 for mutation method 1 and method 3 is very similar, but the fitness value
for the method 2 performed the worst. This indicates that analyzing a
relatively small parameter set in the EXAFS application would require a large
mutation rate to enhance the mutation possibilities and ensure the diversity
of the population.

###  4.2 Cut off Analysis for Selecting Scattering Paths

One of the main difficulties in obtaining an accurate fit using GA for EXAFS
analysis lies in determining the number of scattering paths that may
potentially be observed in an EXAFS measurement and selecting from that set
the actual paths required to replicate the experimental results. The list of
most significant paths can be difficult to obtain due to the effectively
infinite number of potential path combinations. On the other hand, there is no
universally optimized fixed set of paths for all applications and it is
important to deselect insignificant scattering paths with low contribution to
the spectra. The analysis tools must be able to handle these cases.

We have developed a process to analyze the contribution from potential
scattering paths and uncover the scattering paths with the largest
contribution to the spectra. We first calculate the integrated area below the
spectrum curve and the contribution from each individual path. We selected the
paths that contribute more than a user-selectable, pre-defined cut off
percentage of these areas (e.g., 1%). In this manner, we obtain a list of
significant paths for further analysis.

Fig. 4 shows $\chi^{2}$, from Eq. 4, as a function of the cut off percentage.
We analyzed four data sets of copper K-edge EXAFS spectra from metallic copper
collected at temperatures of 10K, 50K, 150K and 298K to test the path
selection method based on the cut off area percentage. The experimental data
were obtained from XASLAB. The initial number of scattering paths that we
employed was 42 which represented a full set of scattering paths with
distances from 2.5527 Å to 7.6580 Å. These paths were used to fit the
experimental data over the K-space range from 3 $\AA^{-1}$ to 17.0 $\AA^{-1}$.
After performing the cut off calculation, a subsequent optimized set of paths
was obtained and applied to a second round of GA optimization that only used
the optimized scattering path list and excluded all paths with insignificant
contributions to the measured spectrum.

The results of utilizing the cut off can be evaluated by using the final
fitness value score (i.e., $\chi^{2}$). However, we must strike a balance
between the number of scattering paths and the accuracy of the final fitness
value. We tested our algorithm by using multiple cut off percentages to
observe the effects on the final average fitness value. We selected seven
different cut off percentages: 10%, 5%, 2%, 1%, 0.67%, 0.5% and 0.3%.

We observed that when the cut off percentage decreased from 10% to 1%, the
average $\chi^{2}$ score decreased (indicating improvement) for all of
temperatures. When the cut off percentage went below 1%, the $\chi^{2}$ score
tended to increase due to the increased number of paths that were included in
the calculation. The increased number of paths led to insignificant changes in
the overall fitness score. In our experiments, we found that the best cut-off
ratio was in the range of 1%, although we allow users to select other cut off
values for their applications.

###  4.3 Computational Performance

It is worth noting that there is no emerging need to parallelize our GA code
since the same data set must be run/analyzed multiple times (e.g., 100) to
gauge the errors compared to the experiments. However, parallelization is
applicable if the number of IO operations is reduced significantly. In our
current GA code, evolutionary operators perform very frequent IO operations.

We measured the scalability of our GA code in terms of the number of
scattering paths. There are two major factors which can affect the complexity
of the computation. The first is the number of sample points in the spectra,
which is usually determined by the experimental conditions of the measurement,
or instrumentation setting. The second is the number of scattering paths
selected by users to represent the exploration range of atomic structure of
interests. Fig. 5 shows the average time spent per generation as a function of
the number of selected paths. We can see that the algorithms scales very well,
with a time complexity $\mathcal{O}(n)$.

Figure 4: $\chi^{2}$ as a function of the cut off percentage for Cu foil EXAFS
Spectra in K-edge at various temperatures.  Figure 5: The complexity increases
as the number of scattering paths increases.

##  5 Conclusion

We have developed a GA based software with the capability of efficiently
performing automated materials characterization analysis of both complex X-ray
spectra and nanoindentation data. We provided multiple crossover and mutation
options in the code from which users can choose to optimize the analysis for
their specific materials systems under study. We have extensively tested our
software under various synthetic and experimental conditions. Our results
demonstrated good scores of fitting without human inputs. We also tested the
code with various cut off percentage to obtain the best scattering paths set.
We note that caution must be exercised in selecting the data set over which to
perform the cut off analysis. It must be representative of the entire
collection of data to be analyzed for the results to be meaningful. The
extensibility of our code base is a major advantage. Adding new analysis
techniques does not require any new debugging of the core module of the GA
code.

##  6 Acknowledgments

This work was supported by author ML’s startup fund. It is also supported
through Idaho National Laboratory (INL) Laboratory Directed Research and
Development (LDRD) Program under DOE Idaho Operations Office Contract DE-
AC07-05ID14517. This research made use of the resources of the High
Performance Computing Center at INL, which is supported by the Office of
Nuclear Energy of the U.S. Department of Energy and the Nuclear Science User
Facilities under Contract No. DE-AC07-05ID14517.

## References

  * [1] Ben Blaiszik et al. “Data automation at light sources” In _AIP Conference Proceedings_ 2054, 2019, pp. 020003
  * [2] George H. Major, Tahereh G. Avval, Behnam Moeini and al. “Assessment of the frequency and nature of erroneous X-ray photoelectron spectroscopy analyses in the scientific literature” In _Journal of Vacuum Science & Technology A_ 38.6, 2020, pp. 061204
  * [3] Jeff Terry, Miu Lun Lau, Jiateng Sun and al. “Analysis of extended X-ray absorption fine structure (EXAFS) data using artificial intelligence techniques” In _Applied Surface Science_ 547, 2021, pp. 149059 URL: https://www.sciencedirect.com/science/article/pii/S0169433221001355
  * [4] “EXAFS Neo”, https://github.com/laumiulun/EXAFS-Neo-Public
  * [5] Matthew Newville “Larch: an analysis package for XAFS and related spectroscopies” In _J Phys Conf Ser_ 430, 2013, pp. 012007
  * [6] John J. Rehr et al. “Parameter-free calculations of X-ray spectra with FEFF9” In _Physical Chemistry Chemical Physics_ 12.21 Royal Society of Chemistry (RSC), 2010, pp. 5503
  * [7] Grant Bunker “Introduction to XAFS” Cambridge University Press, 2009
  * [8] Dale E Sayers, Edward A Stern and Farrel W Lytle “New Technique for investigating noncrystalline structures: Fourier analysis of the extended X-ray absorption fine structure” In _Phys. Rev. Lett._ 27.18 APS, 1971, pp. 1204
  * [9] B. Ravel and M. Newville “ATHENA, ARTEMIS, HEPHAESTUS: data analysis for X-ray absorption spectroscopy using IFEFFIT” In _Journal of Synchrotron Radiation_ 12.4 International Union of Crystallography (IUCr), 2005, pp. 537–541
  * [10] “WinXAS v3.2” Retrieved: 2020-10-30, http://www.winxas.de
  * [11] Grant Bunker, Nicholas Dimakis and Gocha Khelashvili “New methods for EXAFS analysis in structural genomics” In _Journal of Synchrotron Radiation_ 12.1 International Union of Crystallography, 2005, pp. 53–56
  * [12] John H Holland “Genetic algorithms” In _Scientific American_ 267.1 JSTOR, 1992, pp. 66–73
  * [13] David E Goldberg and John H Holland “Genetic Algorithms and Machine Learning” In _Machine Learning_ 3.2 Springer, 1988, pp. 95–99
  * [14] Agoston E Eiben and James E Smith “Introduction to Evolutionary Computing” Springer, 2003
  * [15] I. Rechenberg “Evolutionsstrategie’94” In _Werkstatt Bionik und Evolutionstechnik_ , 1994
  * [16] Danielle R Redhouse “Uncertainty Quantification of a Genetic Algorithm for Neutron Energy Spectrum Adjustment”, 2017
  * [17] Paul L Meyer “Introductory probability and statistical applications” OxfordIBH Publishing, 1965
  * [18] Philip R Bevington et al. “Data Reduction and error analysis for the physical sciences” In _Computers in Physics_ 7.4 American Institute of Physics, 1993, pp. 415–416
  * [19] Siddhartha Chib and Edward Greenberg “Understanding the metropolis-hastings algorithm” In _The American Statistician_ 49.4, 1995, pp. 327–335


# HyperPose: Camera Pose Localization using Attention Hypernetworks

Ron Ferens, Yosi Keller
Bar Ilan University<EMAIL_ADDRESS>
###### Abstract

In this study, we propose the use of attention hypernetworks in camera pose
localization. The dynamic nature of natural scenes, including changes in
environment, perspective, and lighting, creates an inherent domain gap between
the training and test sets that limits the accuracy of contemporary
localization networks. To overcome this issue, we suggest a camera pose
regressor that integrates a hypernetwork. During inference, the hypernetwork
generates adaptive weights for the localization regression heads based on the
input image, effectively reducing the domain gap. We also suggest the use of a
Transformer-Encoder as the hypernetwork, instead of the common multilayer
perceptron, to derive an attention hypernetwork. The proposed approach
achieves superior results compared to state-of-the-art methods on contemporary
datasets. To the best of our knowledge, this is the first instance of using
hypernetworks in camera pose regression, as well as using Transformer-Encoders
as hypernetworks. We make our code publicly
available111https://anonymous.4open.science/r/hyperpose-2023/.

##  1 Introduction

In numerous computer vision applications such as indoor navigation, augmented
reality, and autonomous driving, it is essential to determine the position and
orientation of a camera using a query image. Current camera localization
techniques show various trade-offs between accuracy, runtime, and memory
usage. The most precise methods [2, 3, 28, 35, 27] involve hierarchical
localization pipelines, which commonly use a coarse-to-fine processing flow
that begins with image retrieval to perform an initial localization based on
images similar to the query image, followed by local feature extraction and
matching. To estimate the camera’s 6-Degrees-Of-Freedom (6DoF) accurately, the
resulting 2D-2D matches are mapped to 3D correspondences in the scene’s point
cloud and then used to determine the camera pose through Perspective-n-Point
(PnP) and RANSAC. Another approach, absolute pose regressors (APRs), can
estimate the camera pose with a single forward pass using only the query
image, resulting in significantly lower latency, although being notably less
accurate. Additionally, due to their low memory footprint, APRs can be
deployed as a standalone application on edge devices with limited
computational resources. For a survey of visual camera pose localization,
refer to [44, 30].

Figure 1: Camera pose localization using hypernetworks. (a) A hypernetwork
consisting of a multilayer perceptron computing the weights of the regression
head. (b) The regression weights are computed by an attention-hypernetwork
consisting of a Transformer-Encoder.

PoseNet by Kendall et al. [17] was the first APR approach, using a
convolutional backbone and a multilayer perceptron (MLP) head to regress the
camera’s position and orientation. This approach was simple and able to run at
5ms leading to a multitude of APR methods that improve accuracy by modifying
the backbone and MLP architectures [21, 22, 40, 42, 31, 41, 6] as well as
alternative loss functions and optimization strategies [15, 16, 30]. Indoor
and outdoor environments present dynamic scenarios characterized by varying
lighting conditions, viewing angles, and moving objects. Thus, fitting a
single, global set of parameters to these changing circumstances might limit
the network’s adaptability and result in reduced accuracy.

Hypernetworks [14] are a deep learning architecture where an auxiliary
network, the hypernetwork, generates the weights for the primary network (main
network) based on the current input and the downstream task. The hypernetwork
allows for dynamic and adaptive weight modulation of the main network, leading
to improved flexibility in response to changing conditions [23, 9]. Both the
main network and the hypernetwork are trained in a unified, end-to-end manner
through the use of backpropagation-based optimization.

Our approach differs from traditional Absolute Pose Regression (APR) methods
that rely on a single global image encoding for estimating the 6DoF pose of a
camera [17, 15, 40, 31, 6, 32]. Instead, we use a dual attention-based
approach that simultaneously regresses both translation and orientation,
providing an adaptive analysis of image content. The spatial content of images
contains both informative and uninformative cues, that are effectively handled
by our approach. Furthermore, we introduce a novel attention hypernetwork
formulation that utilizes Transformer-Encoders as hypernetworks to output
coefficients for regression heads based on the input query image.
Specifically, the camera pose regression is divided into two separate
sequence-to-one problems. Intermediate activation maps are processed by two
separate Transformer-Encoders for translation and rotation, as illustrated in
Figure 2. The activation maps are also fed into the relevant attention
hypernetwork, which computes the weights for the localization and orientation
heads. These predicted weights, combined with the output from the encoders,
allow us to perform the regression of the corresponding localization
attributes. This approach allows for an adaptive analysis of the image
content, enabling the regression weights to focus on informative features for
each attribute, rather than relying on a single global image representation.
The use of Transformer-Encoders as hypernetworks is a novel formulation that
provides improved performance in the regression tasks, compared to traditional
fully-connected layers or other neural network architectures. The proposed
dual attention-based architecture results in a more robust and accurate model
for camera pose regression.

In summary, our research offers three key contributions:

  * •

An absolute pose regression approach using hypernetworks that is
experimentally shown to adapt to varying appearance and lighting conditions.

  * •

We propose to use a Transformer-Encoder, instead of the common MLP, as an
attention hypernetwork to learn the regression head parameters.

  * •

The proposed scheme achieves new state-of-the-art APR accuracy across outdoor
and indoor localization benchmarks.

##  2 Related Work

###  2.1 Camera Pose

Camera pose estimation techniques differ by their input during inference and
algorithmic attributes.

3D-based or structure-based localization methods are considered to be the most
accurate for camera pose estimation and have demonstrated state-of-the-art
performance on leading benchmarks. These methods utilize correspondences
between 2D features in an image and 3D world coordinates. A two-phase approach
was introduced by [12, 27, 24, 25] for hierarchical pose estimation pipelines.
In this process, each query image is first encoded using a CNN trained for
image retrieval (IR), based on a pre-mapped image dataset. Tentative
correspondences are then estimated by matching local image features to 3D
matches. The resulting matches are then used by PnP-RANSAC to estimate the
camera pose. DSAC [2] and its follow-up scheme, DSAC++ [3], utilize a CNN
architecture to directly estimate the 3D coordinates from 2D positions in an
image and only require the query image during inference. The same as in
Absolute Pose Regression (APR) methods, separate models must be trained for
each scene, achieving state-of-the-art accuracy.

Relative Pose Regression (RPR) algorithms estimate the relative pose between
an input query image and a set of reference images based on their known
location. Calculating the relative pose involves the estimation of the
relative position and orientation between the query image and the reference
images [47, 4]. An Image Retrieval (IR) approach is utilized to determine the
closest set of neighbor images, and the relative motion is then estimated
between the query image and each of the retrieved images. The estimated
relative poses are utilized, along with the known position of the reference
images, to estimate the camera’s absolute pose. Similarly to 3D and
structured-based localization methods, RPRs involve a multistep process and
require access to a database of images with labeled poses during the inference
phase.

Absolute Pose Regression (APR) directly estimate the camera pose given the
input query image. APRs were first introduced by Kendall [17], using a
modified version of the truncated GoogLeNet architecture, where the Softmax
classification layer was replaced with a series of fully connected layers that
output the pose estimate. Although such approaches are less accurate than
classical structure-based techniques, they offer several advantages, such as
faster inference times (milliseconds), reduced memory footprint (megabytes),
and do not require feature engineering. Variations in the encoder and MLP
architectures have been proposed to enhance APR accuracy. Furthermore,
modifications to the loss function have been suggested to improve the balance
between orientation and position objectives [16] or to incorporate other types
of information [5]. For an in-depth examination of the various architectural
designs and loss functions utilized in camera pose regression, please refer to
[30].

Figure 2: The proposed method consists of a dual-branch structure in the main
network and an accompanying hypernetwork. The Transformer-Encoders in the main
network sample activation maps generated by the convolutional backbone. In
parallel, the hypernetwork generates a set of weights for the regression head
of the main network, based on the input query image. The resulting adaptive
weights, combined with the generalized latent vectors produced by each encoder
in the main network, are utilized to estimate the camera pose components,
either position (x) or orientation (q).

###  2.2 Transformers

Originally developed for machine translation [38], Transformers have since
become a state-of-the-art architecture in natural language processing. Recent
studies have also shown the success of using Transformers to encode visual
data [11]. Our work builds upon these studies by demonstrating the
effectiveness of Transformers in incorporating the spatial information needed
for accurate pose localization. In particular, to the best of our knowledge,
we are the first to propose Transformer-Encoders as an efficient hypernetwork
that adapts the weights of the regression heads based on the input query
image.

###  2.3 Hypernetworks

Hypernetworks were first presented by Ha et al. [14]. These are neural
networks that have been designed to predict the weights of a primary network.
The primary network’s weights can be adjusted based on specific inputs,
resulting in a more expressive and adaptive model. Hypernetworks have been
used in various applications such as learning implicit neural representations
[9], semantic segmentation [23], 3D scene representation and modeling [19, 33,
34] and continuous learning [39], to name a few.

##  3 Camera Pose Localization using Attention Hypernetworks

The proposed localization network consists of two main parts: the main network
(APR), which is designed to estimate the camera’s absolute position, and the
hypernetwork, which predicts the regression weights of the main network based
on the input query image. The overall network is illustrated in Fig. 2 where
the APR performs a forward pass on the input image to localize the capture
camera. The camera pose is represented as the tuple $<\mathbf{x},\mathbf{q}>$,
where $\mathbf{x}\in\mathbb{R}^{3}$ is the camera’s position in world
coordinates and $\mathbf{q}\in\mathbb{R}^{4}$ is the quaternion encoding of
its 3D orientation. Our method utilizes separate positional and orientational
Transformer-Encoders to effectively integrate the flattened intermediate
activation maps generated by a convolutional backbone. Following [8], we
formulate the hypernetwork $H$ as a parametric function
$\mathbf{\theta}{{}_{x}}=H(\mathbf{\theta},\mathbf{x})$, where
$\mathbf{\theta}$ represents the parameters of the hypernetwork, $\mathbf{x}$
the input activation maps, and $\mathbf{\theta}{{}_{x}}$ the parameters of the
regression heads layer for the image $\mathbf{x}$.

###  3.1 Network Architecture

In line with Fig. 2, the input query image $I_{q}$ is processed by a
convolutional backbone that calculates the intermediate activation maps of two
different layers. These activation maps are then utilized as inputs by both
the main network’s and hypernetwork’s Transformer-Encoders. The hypernetwork’s
encoders convert the activation maps into latent vectors through their
processing. These latent vectors are then passed through multilayer
perceptrons (MLPs) to generate the weights for the translation and orientation
regression heads. Simultaneously, the positional and orientational encoders
within the main network process the activation maps and integrate sequential
representations into single latent vectors. These latent vectors are used to
estimate camera position and orientation, using the weights
$\mathbf{\theta}{{}_{x}}$ generated by the hypernetwork $H$.

####  3.1.1 Main Network

The main network in the HyperPose architecture is composed of a convolutional
backbone and two separate branches for regressing the camera’s position and
orientation. Each branch includes its own Transformer-Encoder and a multilayer
perceptron (MLP) head. Studies have demonstrated that substituting the
patchify stem, utilized in early visual processing of a Vision Transformer
(ViT), with a conventional convolutional stem enhances optimization stability
and elevates peak performance to a higher level [43]. Hence, given an input
query image $I_{q}\in\mathbb{R}^{H_{q}\times W_{q}\times C_{q}}$, a shared
convolutional backbone generates activation maps. Two different activation
maps are selected from the CNN based on the particular learned task. The
activation maps, represented as $M\in\mathbb{R}^{H_{M}\times W_{M}\times
C_{M}^{In}}$, are processed and transformed into
$\tilde{M}\in\mathbb{R}^{H_{M}\times W_{M}\times C_{M}^{Out}}$ using a
$1\times 1$ convolutional layer, following the sequence preparation procedure
described in [38]. To maintain the spatial relationships between the cells of
the activation map, positional encoding is applied. The processed sequence is
then augmented with a learned orientation/position token
$T_{m}\in\mathbb{R}^{C_{M}^{Out}}$, creating a concatenated sequence that
serves as the input to the task-specific Transformer Encoder [10]:

$E_{m}^{In}=[T_{m},\tilde{M}]\in\mathbb{R}^{(H_{M}W_{M}=1)\times
C_{M}^{Out}}.$ (1)

The Transformer-Encoder consists of $N$ blocks, all of which are identical in
design. Each block is composed of a multi-head attention (MHA) mechanism for
self-attention and a two-layer MLP with Swish activation functions [26] as
specified in [7]. The input is subjected to Layer Normalization (LN) prior to
processing by the MHA and MLP modules, as per [38]. The outputs of the MHA and
MLP are then combined with the input via residual connections and dropout,
following the approach outlined in [32]. As described in [11], position
encoding is added to the input before each layer, and the final output
undergoes additional LN normalization. The output at the position of the
special token $E_{m_{0}}^{Out}\in\mathbb{R}^{C_{M}^{Out}}$, represents a
context-aware and global summarization of the local features in the input
activation map. Finally, based on the MLP weights output by the hypernetwork,
the estimated camera position and orientation is regressed using the global
summarized latent vector $E_{0}^{Out}$.

####  3.1.2 Hypernetwork

The hypernetwork, similar in design to the main network, consists of two
distinct branches, each responsible for the regression of the weights for the
regression position and orientation heads in the main network. Through the
application of an identical preprocessing flow, a sequence is generated for
the input of the Transformers encoders in the hypernetwork, derived from the
encoded activation maps and learned tokens $T_{h}\in\mathbb{R}^{C_{M}^{Out}}$
[10]:

$E_{h}^{In}=[T_{h},\tilde{M}]\in\mathbb{R}^{(H_{M}W_{M}=1)\times
C_{M}^{Out}}.$ (2)

The global summarized latent vector from the encoders,
$E_{h_{0}}^{Out}\in\mathbb{R}^{C_{M}^{Out}}$, is processed by multiple
multilayer perceptrons (MLPs) followed by Swish activation functions, each of
which is tasked with estimating the weights of a single corresponding layer in
the regression heads of the main network:

$\theta_{h}^{{Out}_{i}}=H(\theta_{h}^{{Out}_{i}},E_{h_{0}}^{Out})\in\mathbb{R}^{C_{M}^{Out}\times
C_{H}^{{Out}_{i}}},$ (3)

where $i$ is the index of the regression head layer.

###  3.2 Optimization Criteria

The localization error is calculated by comparing the translation position and
orientation of the ground truth pose
$\mathbf{p}_{gt}=<\mathbf{x}_{gt},\mathbf{q}_{gt}>$, where
$\mathbf{x}\in\mathbb{R}^{3}$ represents the position of the camera in the
world and $\mathbf{q}\in\mathbb{R}^{4}$ denotes its orientation encoded as a
quaternion, with the estimated pose
$\mathbf{p}_{est}=<\mathbf{x}_{est},\mathbf{q}_{est}>$. The translation error,
represented by $L_{x}$, is commonly expressed in meters and is calculated as
the Euclidean distance between the ground truth and the estimated camera
positions. The orientation error, $L_{q}$, is defined as the Euclidean
distance between the ground truth and the estimated unit vector quaternions
[30]:

$L_{x}=||\mathbf{x}_{0}-\mathbf{x}||_{2},$ (4)
$L_{q}=\left\|\mathbf{q_{0}}-\frac{\mathbf{q}}{||\mathbf{q}||}\right\|_{2}$
(5)

where $\mathbf{q}$ is a normalized norm quaternion to ensure it is a valid
orientation encoding.

As a multi-regression optimization problem, we adopt the method outlined in
[16] by using different weighting schemes, based on the uncertainty of each
task, to balance the individual loss functions defined for the separate
objectives. The model optimizes the aggregated loss function, which is a
combination of the weighted individual losses.

$L_{p}=L_{x}\exp(-s_{x})+s_{x}+L_{q}\exp(-s_{q})+s_{q},$ (6)

where $s_{x}$ and $s_{q}$ are the learned parameters. As a further
optimization, we adopt the approach detailed in [31] and follow a three-step
training process. Firstly, the entire network is trained using the aggregated
loss described in Eq. 6. In the subsequent two steps, each MLP head is fine-
tuned independently using its specific loss.

###  3.3 Implementation Details

In our shared convolutional backbone architecture, we employed an
EfficientNet-B0 [36]. This model outperforms the accuracy of most existing
convolutional neural networks and boasts improved efficiency. It consists of
multiple levels of reduction, each representing activation maps with a reduced
spatial resolution and an increased network depth. We used the activation maps
of two endpoints (reduction levels) as input for our position and orientation
branches: $m_{rdct4}\in\mathbb{R}^{14\times 14\times 112}$ and
$m_{rdct3}\in\mathbb{R}^{28\times 28\times 40}$, respectively. According to
the method described in [32], we applied linear projections on each activation
map to a common depth dimension of $C_{M}^{Out}=256$ and learned the
positional encoding of the same depth. Our implementation of the EfficientNet
model is based on an open-source version [20]. In both the main network and
the hypernetwork, we incorporated six-layer Transformer Encoders in each
branch (position and orientation). Each block consists of a MHA layer with
four heads, followed by an MLP that preserves the input dimension. A dropout
rate of $p=0.1$ was applied to each MLP layer. The output dimension of the
Transformer-Encoders in the main network was set to $C_{M}^{Out}=256$, while
in the hypernetwork it was increased to $C_{H}^{Out}=512$. The position and
orientation branches of the hypernetwork generate the weights of three
regression layers: input, hidden, and output. The number of weights in the
position and orientation branches differ. The hypernetwork’s position branch
produces the weights for an MLP with dimensions 512, 256, and 3, while the
orientation branch generates weights for an MLP with dimensions 512, 512, and
4. The weights generated by the hypernetwork are then incorporated into the
regression heads of the main network to estimate the position and orientation
of the input query image. The input and hidden layers of the regression head
utilize the Swish activation function.

##  4 Experimental Results

###  4.1 Datasets

The experimental evaluation was conducted using the 7 Scenes [13] and
Cambridge Landmarks [17] APR evaluation benchmark datasets. The Cambridge
Landmarks dataset represents an urban outdoor environment with a spatial
extent ranging from approximately $900$ to $5500$ square meters. We present
results for four of its six scenes, as the remaining two scenes have limited
coverage in the existing literature or are reported to have possible faults.
The 7 Scenes dataset consists of seven small-scale indoor scenes with a
spatial extent ranging from approximately $1$ to $10$ square meters. These
datasets present various challenges commonly encountered in visual
localization applications, including differences in scale, indoor/outdoor
environments, repetitive elements, textureless features, significant viewpoint
changes, and differences in trajectory between training and test sets.

###  4.2 Training Details

Table 1: Comparison to state-of-the-art methods: Cambridge Landmarks. We
report the median position/orientation error in meters/degrees for each
method. The first and second most effective APRs are distinguished using Red
and Blue colors, respectively.

|  | College | Hospital | Shop Facade | St. Mary | Avg.
---|---|---|---|---|---|---
| DSAC [4] | 0.18,0.3° | 0.21,0.4° | 0.05,0.3° | 0.15,0.5° | 0.15,0.4°
Seq. | MapNet [5] | 1.08,1.9° | 1.94,3.9° | 1.49,4.2° | 2.00,4.5° | 1.63,3.6°
GL-Net [46] | 0.59,0.7° | 1.88,2.8° | 0.50,2.9° | 1.90,3.3° | 1.22,2.4°
IR | VLAD [37] | 2.80,5.7° | 4.01,7.1° | 1.11,7.6° | 2.31,8.0° | 2.56,7.1°
VLAD+Inter [29] | 1.48,4.5° | 2.68,4.6° | 0.90,4.3° | 1.62,6.1° | 1.67,4.9°
RPR | EssNet [47] | 0.76,1.9° | 1.39,2.8° | 0.84,4.3° | 1.32,4.7° | 1.08,3.4°
NC-EssNet [47] | 0.61,1.6° | 0.95,2.7° | 0.7,3.4° | 1.12,3.6° | 0.85,2.8°
RelocGNN[4] | 0.48,1.0° | 1.14,2.5° | 0.48,2.5° | 1.52,3.2° | 0.91,2.3°
APR | PoseNet [17] | 1.92,5.40° | 2.31,5.38° | 1.46,8.08° | 2.65,8.48° | 2.08,6.83°
BayesianPN [15] | 1.74,4.06° | 2.57,5.14° | 1.25,7.54° | 2.11,8.38° | 1.91,6.28°
LSTM-PN [40] | 0.99,3.65° | 1.51,4.29° | 1.18,7.44° | 1.52,6.68° | 1.30,5.57°
SVS-Pose [22] | 1.06,2.81° |  1.50,4.03° |  0.63,5.73° | 2.11,8.11° | 1.32,5.17°
GPoseNet [6] | 1.61,2.29° | 2.62,3.89° | 1.14,5.73° | 2.93,6.46° | 2.07,4.59°
PoseNetLearn [16] | 0.99,1.06° | 2.17,2.94° | 1.05,3.97° | 1.49,3.43° | 1.42,2.85°
GeoPoseNet [16] | 0.88,1.04° | 3.20,3.29° | 0.88,3.78° | 1.57,3.32° | 1.63,2.86°
MapNet [5] | 1.07,1.89° | 1.94,3.91° | 1.49,4.22° | 2.00,4.53° | 1.62,3.64°
IRPNet [31] | 1.18,2.19° | 1.87,3.38° | 0.72,3.47° | 1.87,4.94° | 1.41,3.50°
MS-Trans-1S[32] |  0.72,2.55° | 2.07,3.24° | 0.68,3.66° | 1.10,5.26° |  1.14,3.68°
HyperPose (Ours) |  0.56,2.40° |  1.41,2.91° |  0.54,3.37° |  0.98,4.86° |  0.87,3.43°

222https://anonymous.4open.science/r/hyperpose-2023/

The training of the network is executed in three phases. During the first
phase, all network components are trained simultaneously. In the second phase,
only the translation-related layers, such as the hypernetwork and regression
layers, are fine-tuned while other parts of the network are kept frozen. In
the final phase, the orientation-related components are fine-tuned to achieve
improved performance without compromising the localization objectives. The
Adam optimization algorithm is used to minimize the loss function with
$\beta_{1}=0.9,\beta_{2}=0.999$, and $\epsilon=10^{-10}$ as the
hyperparameters. The initial values of the loss parameters are established
based on the characteristics of the dataset utilized. The batch size is set to
$8$, and the initial learning rate is $\lambda=10^{-4}$. The learning rate
decreases by $25\%$ every 100 epochs for indoor localization and every 200
epochs for outdoor localization, with a maximum of $300$ and $1000$ epochs,
respectively. Additionally, a weight decay of $10^{-4}$ and a dropout of
$p=0.1$ are applied to the encoders during training. Our model is subjected to
augmentation procedures to enhance its generalization, in line with the
approach used in [17]. During training, the image is resized so that its
smaller edge is resized to $256$ pixels and a random $224\times 224$ crop is
taken. Random adjustments are made to the brightness, contrast, and saturation
of the image. During testing, the image is rescaled with the smaller edge
resized to $256$ pixels, and a center crop is taken without any further
augmentation. The models were trained on multiple NVIDIA $2080Ti$ GPUs and the
PyTorch framework.

Table 2: Comparison with state-of-the-art methods: 7Scenes. We report the median position/orientation error in meters/degrees for each method. The first and second most effective APRs are distinguished using Red and Blue colors, respectively. |  | Chess | Fire | Heads | Office | Pumpkin | Kitchen | Stairs
---|---|---|---|---|---|---|---|---
| DSAC* [4] | 0.02,1.1° | 0.02,1.0° | 0.01,1.8° | 0.03,1.2° | 0.04,1.4° | 0.03,1.7° | 0.04,1.4°
Seq. | LsG [45] | 0.09,3.3° | 0.26,10.9° | 0.17,12.7° | 0.18,5.5° | 0.20,3.7° | 0.23,4.9° | 0.23,11.3°
MapNet[5] | 0.08,3.3° | 0.27,11.7° | 0.18,13.3° | 0.17,5.2° | 0.22,4.0° | 0.23,4.9° | 0.30,12.1°
GL-Net[46] | 0.08,2.8° | 0.26,8.9° | 0.17,11.4° | 0.18,13.3° | 0.15,2.8° | 0.25,4.5° | 0.23,8.8°
IR | VLAD [37] | 0.21,12.5° | 0.33,13.8° | 0.15,14.9° | 0.28,11.2° | 0.31,11.2° | 0.30,11.3° | 0.25,12.3°
VLAD+Inter[29] | 0.18,10.0° | 0.33,12.4° | 0.14,14.3° | 0.25,10.1° | 0.26,9.4° | 0.27,11.1° | 0.24,14.7°
RPR | NN-Net [18] | 0.13,6.5° | 0.26,12.7° | 0.14,12.3° | 0.21,7.4° | 0.24,6.4° | 0.24,8.0° | 0.27,11.8°
RelocNet[1] | 0.12,4.1° | 0.26,10.4° | 0.14,10.5° | 0.18,5.3° | 0.26,4.2° | 0.23,5.1° | 0.28,7.5°
EssNet [47] | 0.13,5.1° | 0.27,10.1° | 0.15,9.9° | 0.21,6.9° | 0.22,6.1° | 0.23,6.9° | 0.32,11.2°
NC-EssNet [47] | 0.12,5.6° | 0.26,9.6° | 0.14,10.7° | 0.20,6.7° | 0.22,5.7° | 0.22,6.3° | 0.31,7.9°
RelocGNN[4] | 0.08,2.7° | 0.21,7.5° | 0.13,8.70° | 0.15,4.1° | 0.15,3.5° | 0.19,3.7° | 0.22,6.5°
APR | PoseNet [17] | 0.32,8.12° | 0.47,14.4° | 0.29,12.0° | 0.48,7.68° | 0.47,8.42° | 0.59,8.64° | 0.47,13.8°
BayesianPN [15] | 0.37,7.24° | 0.43,13.7° | 0.31,12.0° | 0.48,8.04° | 0.61,7.08° | 0.58,7.54° | 0.48,13.1°
LSTM-PN [40] | 0.24,5.77° | 0.34,11.9° | 0.21,13.7° | 0.30,8.08° | 0.33,7.00° | 0.37,8.83° | 0.40,13.7°
GPoseNet [6] | 0.20,7.11° | 0.38,12.3° | 0.21,13.8° | 0.28,8.83° | 0.37,6.94° | 0.35,8.15° | 0.37,12.5°
PoseNetLearn[16] | 0.14,4.50° | 0.27,11.8° | 0.18,12.1° | 0.20,5.77° | 0.25,4.82° | 0.24,5.52° | 0.37,10.6°
GeoPoseNet[16] | 0.13,4.48° | 0.27,11.3° | 0.17,13.0° |  0.19,5.55° | 0.26,4.75° | 0.23,5.35° | 0.35,12.4°
MapNet[5] |  0.08,3.25° | 0.27,11.7° | 0.18,13.3° |  0.17,5.15° | 0.22,4.02° | 0.23,4.93° | 0.30,12.1°
IRPNet[31] | 0.13,5.64° | 0.25,9.67° |  0.15,13.1° | 0.24,6.33° | 0.22,5.78° | 0.30,7.29° | 0.34,11.6°
AttLoc[41] |  0.10,4.07° | 0.25,11.4° | 0.16,11.8° |  0.17,5.34° | 0.21,4.37° | 0.23,5.42° |  0.26,10.5°
MS-Trans-1S[32] |  0.10,5.86° |  0.23,11.0° |  0.15,12.8° |  0.17,6.56° |  0.18,5.32° |  0.17,6.30° |  0.26,11.3°
HyperPose (Ours) |  0.08,6.29° |  0.22,11.2° |  0.11,12.7° |  0.17,7.53° |  0.16,6.66° |  0.17, 8.48° |  0.26,10.8°

###  4.3 Results

The primary objective of the proposed scheme is to improve the accuracy of
absolute pose regression. To evaluate its performance, Tables 1 and 2 compare
the median position error, measured in meters, and the orientation error,
measured in degrees, of the proposed method with recent state-of-the-art
localization algorithms on the Cambridge Landmarks and 7 Scenes datasets,
respectively. For both indoor and outdoor datasets, in addition to the results
of leading APRs, we report the median position and orientation errors of
structure-based, sequence-based, IR and RPR methods. Although DSAC [4]
provides SOTA accuracy, our proposed method outperforms all other APRs
methods. HyperPose achieves the lowest position estimation error in all scenes
in both the Cambridge outdoor dataset and the 7Scenes indoor dataset, with the
most accurate overall average of $0.87m$ and $0.17m$, respectively.
Furthermore, HyperPose demonstrates the best orientational estimation error on
both the Hospital and ShopFace scenes, in the Cambridge dataset.

###  4.4 Hypernetwork Output Weights

(a) Positional clusters (b) Hypernetworks weights clusters

Figure 3: Clusters of the Shop Facade test set images, that are part of the
Cambridge Landmarks dataset. The clusters are computed using K-Means with
$K=4$. (a) Clustering the corresponding camera (X, Y) positions. (b)
Clustering the 2D t-SNE projections of the regression weights computed by the
hypernetwork.

Given an input query image $I_{q}$, hypernetworks generate weights for the
position and orientation regression heads, enabling adaptation of the
network’s output based on the visual features present in the image. The scene
depicted in the image is likely to exhibit a range of attributes, such as
varying illumination conditions, different viewpoints, dynamic objects, and
changing backgrounds. As a result, it is expected that the generated weights
will be adapted to accommodate these fluctuations, so that the images from the
same scene (location) will be jointly clustered despite the appearance
variations, as shown in Fig. 3. Figure 3(a) depicts the clustering of the Shop
Facade scene test set from the Cambridge Landmarks dataset. The spatial (X,Y)
coordinates were clustered using K-Means with $K=4$. The images are colored
according to their cluster index. As expected, the clusters were formed based
on the proximity and location of the images within the scene’s coordinates.
Figure 2(b) shows the K-Means clustering of the 2D t-SNE projections of the
regression weights generated by the hypernetwork. Figure 3(b) shows that the
images are grouped based on their visual attributes and viewing angles, which
demonstrates that the hypernetwork adapts its output to the input image. This
differs from other state-of-the-art APR solutions, where the regression head
weights are static and learned during the training phase, thus restricting
their ability to adapt to the variations in input images.

###  4.5 Ablation Study

In order to gauge the performance of the chosen architecture, we performed an
ablation study. The comparison was based on the model’s performance on the
ShopFacade scene in the Cambridge Landmarks dataset, measured at the end of
the initial training phase. For each ablation, we changed and studied a single
scheme parameter.

Residual hypernetwork output. Table 3 presents the results obtained by
regressing both the absolute pose and the residual additive branch, to support
the reasoning behind the configuration of the hypernetworks to output the
residual position and orientation. The residual-based architecture outperforms
the direct regression approach, as evidenced by the lower errors in both
position and orientation estimation achieved by the former.

Table 3: Ablation study of the network residual architecture. We report the median position and orientation errors. Architecture | Position | Orientation
---|---|---
| [meters] | [degrees]
Direct Regression | 0.55 | 4.06
Residual Regression | 0.53 | 3.55

Hypernetwork embedding dimension. We assess the effect of altering the size of
the fully-connected layers in both the position and orientation branches of
the regression heads. The results of our experiment are shown in Table 4,
where we present the results obtained by varying the dimensions of the layers.

Table 4: Ablation study of the embedding dimension regressed by the hypernetworks. We report the median position and orientation errors. Embedding Dimensions | Position | Orientation
---|---|---
[Position/Orientation] | [meters] | [degrees]
256/256 | 0.66 | 4.3
256/512 | 0.53 | 3.55
512/512 | 0.68 | 4.33

Hypernetwork architecture. We evaluated using an alternative conventional MLP-
based hypernetwork design. This design computes the regression layer weights
using an MLP, whose input is the terminal latent vector of the
EnfficientNet-B0 backbone. The proposed attention-hypernetwork architecture
exhibits superior performance in estimating both positional and orientational
errors. A plausible explanation is that spatial information is already
condensed into a vector by the output of the backbone network, which
constrains the MLP-based hypernetwork branches from addressing finer details
in the activation maps.

Table 5: Ablations hypernetwork architecture. We report the median position and orientation errors. Architecture | Position | Orientation
---|---|---
| [meters] | [degrees]
Backbone embedding | 0.68 | 4.29
Transformer-Encoders | 0.57 | 4.06

Hypernetwork output layers. By design, the output of hypernetworks determines
the number of regression layers in the regression heads of the main network.
Table 6 details the position estimation error vs. the number of layers.

Table 6: Ablations of the hypernetwork number of output weights. We report the median position and orientation errors. Regressed Layers | Position | Orientation
---|---|---
| [meters] | [degrees]
Single Layer | 0.64 | 4.13
Input/Output layers | 0.65 | 3.99
Input/Hidden/Output layers | 0.66 | 3.55

Rotation representations. The authors of [48] contend that 3D and 4D rotation
representations are suboptimal for network regression, whereas continuous
representations in 5D and 6D are more appropriate for learning. As shown in
Table 7, employing a 4D-based representation in conjunction with the
rotational loss specified in Eq. 5 led to the lowest orientational estimation
error. In both the Quaternions and 4D-Norm approaches, the rotation regressor
generates four values that estimate the camera’s orientation (quaternion), but
the latter computes the orientation loss based on rotation matrices instead.

Table 7: Ablation of the 3D rotation encoding. We report the median orientation errors. Rotation representation | Orientation
---|---
| [degrees]
Quaternions | 3.55
4D-Norm [48] | 4.08
6D [48] | 4.54

##  5 Conclusions

In this study, we present a novel attention-based architecture for absolute
pose regression that uses a hypernetwork to estimate the regression head
weights. This allows us to improve localization accuracy by adapting the
regression weights to the input image embedding. To the best of our knowledge,
this is the first use of hypernetworks to have been used in localization in
general, and the first use of Transformer-Encoders in hypernetworks, in
particular. Our approach is shown to compare favorably with contemporary APR
schemes.

## References

  * [1] Vassileios Balntas, Shuda Li, and Victor Prisacariu.  Relocnet: Continuous metric learning relocalisation using neural nets.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [2] E. Brachmann, A. Krull, S. Nowozin, J. Shotton, F. Michel, S. Gumhold, and C. Rother.  DSAC - differentiable RANSAC for camera localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2492–2500, Los Alamitos, CA, USA, jul 2017\.
  * [3] E. Brachmann and C. Rother.  Learning less is more - 6D camera localization via 3D surface regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4654–4662, 2018.
  * [4] E. Brachmann and C. Rother.  Visual camera re-localization from rgb and rgb-d images using dsac.  IEEE Transactions on Pattern Analysis and Machine Intelligence, (01):1–1, apr 2021.
  * [5] Samarth Brahmbhatt, Jinwei Gu, Kihwan Kim, James Hays, and Jan Kautz.  Geometry-aware learning of maps for camera localization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [6] Ming Cai, Chunhua Shen, and Ian Reid.  A hybrid probabilistic model for camera relocalization.  In Proceedings of the British Machine Vision Conference, page 238, 2018.
  * [7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 213–229, Cham, 2020.
  * [8] Yoav Chai, Raja Giryes, and Lior Wolf.  Supervised and unsupervised learning of parameterized color enhancement.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 992–1000, 2020.
  * [9] Yinbo Chen and Xiaolong Wang.  Transformers as meta-learners for implicit neural representations.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 170–187. Springer, 2022.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [12] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler.  D2-net: A trainable cnn for joint description and detection of local features.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8084–8093, 2019.
  * [13] B. Glocker, S. Izadi, J. Shotton, and A. Criminisi.  Real-time rgb-d camera relocalization.  In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 173–179, 2013.
  * [14] David Ha, Andrew Dai, and Quoc V Le.  Hypernetworks.  arXiv preprint arXiv:1609.09106, 2016.
  * [15] Alex Kendall and Roberto Cipolla.  Modelling uncertainty in deep learning for camera relocalization.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 4762–4769, 2016.
  * [16] A. Kendall and R. Cipolla.  Geometric loss functions for camera pose regression with deep learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6555–6564, 2017.
  * [17] A. Kendall, M. Grimes, and R. Cipolla.  Posenet: A convolutional network for real-time 6-dof camera relocalization.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2938–2946, 2015.
  * [18] Z. Laskar, I. Melekhov, S. Kalia, and J. Kannala.  Camera relocalization by computing pairwise relative poses using convolutional neural network.  In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pages 920–929, 2017.
  * [19] Gidi Littwin and Lior Wolf.  Deep meta functionals for shape representation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1824–1833, 2019.
  * [20] Luke Melaskyriazi.  efficientnet-pytorch, 2019.
  * [21] Iaroslav Melekhov, Juha Ylioinas, Juho Kannala, and Esa Rahtu.  Image-based localization using hourglass networks.  In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW), pages 870–877, 2017.
  * [22] Tayyab Naseer and W. Burgard.  Deep regression for monocular camera-based 6-dof global localization in outdoor environments.  2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1525–1530, 2017.
  * [23] Yuval Nirkin, Lior Wolf, and Tal Hassner.  Hyperseg: Patch-wise hypernetwork for real-time semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4061–4070, 2021.
  * [24] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han.  Large-scale image retrieval with attentive deep local features.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3476–3485, 2017.
  * [25] N. Radwan, A. Valada, and W. Burgard.  Vlocnet++: Deep multitask learning for semantic visual localization and odometry.  IEEE Robotics and Automation Letters, 3(4):4407–4414, 2018.
  * [26] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  arXiv preprint arXiv:1710.05941, 2017.
  * [27] P. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12708–12717, 2019.
  * [28] T. Sattler, B. Leibe, and L. Kobbelt.  Efficient effective prioritized matching for large-scale image-based localization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9):1744–1756, 2017.
  * [29] T. Sattler, Q. Zhou, M. Pollefeys, and L. Leal-Taixé.  Understanding the limitations of cnn-based absolute camera pose regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3297–3307, 2019.
  * [30] Yoli Shavit and Ron Ferens.  Introduction to camera pose estimation with deep learning.  In arXiv preprint arXiv:1907.05272, 2019.
  * [31] Yoli Shavit and Ron Ferens.  Do we really need scene-specific pose encoders?  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 3186–3192. IEEE, 2021.
  * [32] Yoli Shavit, Ron Ferens, and Yosi Keller.  Learning multi-scene absolute pose regression with transformers.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
  * [33] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  Advances in Neural Information Processing Systems (NIPS), 33:7462–7473, 2020.
  * [34] Przemysław Spurek, Artur Kasymov, Marcin Mazur, Diana Janik, Slawomir K Tadeja, J Tabor, T Trzciński, et al.  Hyperpocket: Generative point cloud completion.  In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6848–6853. IEEE, 2022.
  * [35] H. Taira, M. Okutomi, T. Sattler, M. Cimpoi, M. Pollefeys, J. Sivic, T. Pajdla, and A. Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2019.
  * [36] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  volume 97 of Proceedings of Machine Learning Research, pages 6105–6114, Long Beach, California, USA, 09–15 Jun 2019.
  * [37] Akihiko Torii, Relja Arandjelovic, Josef Sivic, Masatoshi Okutomi, and Tomas Pajdla.  24/7 place recognition by view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NIPS), volume 30, pages 5998–6008, 2017.
  * [39] Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe.  Continual learning with hypernetworks.  arXiv preprint arXiv:1906.00695, 2019.
  * [40] F. Walch, C. Hazirbas, L. Leal-Taixé, T. Sattler, S. Hilsenbeck, and D. Cremers.  Image-based localization using lstms for structured feature correlation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 627–637, 2017.
  * [41] Bing Wang, Changhao Chen, Chris Xiaoxuan Lu, Peijun Zhao, Niki Trigoni, and Andrew Markham.  Atloc: Attention guided camera localization.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10393–10401, 2020.
  * [42] J. Wu, L. Ma, and X. Hu.  Delving deeper into convolutional neural networks for camera relocalization.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 5644–5651, 2017.
  * [43] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick.  Early convolutions help transformers see better.  Advances in Neural Information Processing Systems (NIPS), 34:30392–30400, 2021.
  * [44] Meng Xu, Youchen Wang, Bin Xu, Jun Zhang, Jian Ren, Stefan Poslad, and Pengfei Xu.  A critical analysis of image-based camera pose estimation techniques.  arXiv preprint arXiv:2201.05816, 2022.
  * [45] Fei Xue, Xin Wang, Zike Yan, Qiuyuan Wang, Junqiu Wang, and Hongbin Zha.  Local supports global: Deep camera relocalization with sequence enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [46] Fei Xue, Xin Wu, Shaojun Cai, and Junqiu Wang.  Learning multi-view camera relocalization with graph neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [47] Qunjie Zhou, Torsten Sattler, Marc Pollefeys, and Laura Leal-Taixe.  To learn or not to learn: Visual localization from essential matrices.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 3319–3326. IEEE, 2020.
  * [48] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5745–5753, 2019.