# AffineGlue: Joint Matching and Robust Estimation

Daniel Barath1, Dmytro Mishkin2,4,
Luca Cavalli1, Paul-Edouard Sarlin1, Petr Hruby1, Marc Pollefeys1,3
1 Department of Computer Science, ETH Zurich, 2 Visual Recognition Group, CTU
in Prague,
3 Microsoft Mixed Reality and AI Zurich Lab, 4 HOVER Inc.
<EMAIL_ADDRESS>
###### Abstract

We propose AffineGlue, a method for joint two-view feature matching and robust
estimation that reduces the combinatorial complexity of the problem by
employing single-point minimal solvers. AffineGlue selects potential matches
from one-to-many correspondences to estimate minimal models. Guided matching
is then used to find matches consistent with the model, suffering less from
the ambiguities of one-to-one matches. Moreover, we derive a new minimal
solver for homography estimation, requiring only a single affine
correspondence (AC) and a gravity prior. Furthermore, we train a neural
network to reject ACs that are unlikely to lead to a good model. AffineGlue is
superior to the SOTA on real-world datasets, even when assuming that the
gravity direction points downwards. On PhotoTourism, the AUC@10∘ score is
improved by 6.6 points compared to the SOTA. On ScanNet, AffineGlue makes
SuperPoint and SuperGlue achieve similar accuracy as the detector-free LoFTR.

Figure 1: The steps of the AffineGlue pipeline are as follows: (1) features
with affine shapes are detected in the input images, _e.g_., by SuperPoint
[30] combined with AffNet [62]. (2) For each feature in the source image, the
matching by, _e.g_. SuperGlue [80], is often ambiguous, especially, at
repeated patterns. Thus, we form one-to-many matches for each point in the
source image. (3) AffineGlue iteratively selects a candidate one-to-one affine
correspondence and estimates the model (_e.g_., relative pose) by a single-
point solver. Guided sampling then forms one-to-one correspondences consistent
with the estimated model to calculate its score and select its inliers.

##  1 Introduction

Matching two or more images of a scene is a fundamental problem in vision with
a wide range of applications, such as image retrieval [56, 2, 72, 91, 69],
structure-from-motion [1, 47, 84, 101, 11], localization [81, 82, 57, 70],
SLAM [35, 66, 29, 31], and multi-view stereo [38, 39, 50, 23]. The traditional
image matching pipeline consists of three main steps – local feature
detection, matching, and geometric robust estimation. Due to this consecutive
nature, matching failures often lead to failure in subsequent geometric robust
estimation, rendering the pipeline unreliable. While recent algorithms [88,
96, 22] perform feature detection and matching jointly, at the cost of
significantly increased run-time for all-pair 3D reconstruction, there is
still a gap in the literature for methods that allow for simultaneous matching
and robust estimation. To address this issue, we propose a novel approach
called AffineGlue that employs joint feature matching and robust estimation by
iteratively selecting potential matches, estimating the model, and performing
guided matching to calculate the model score, _e.g_., via its support. While
most methods need to commit to one-to-one matches to keep the problem
tractable, we relax this to one-to-$k$ matches.

Feature detection and matching. Local features have been and still are the
main workhorse in 3D reconstruction. Traditionally, local features involve
three main steps: (scale-covariant) keypoint detection, orientation
estimation, and descriptor extraction. Keypoint detection is typically
performed on the scale pyramid employing a handcrafted response function, such
as Hessian [17, 60], Harris [43, 60], Difference of Gaussians (DoG [56]), or
learned ones like FAST [78] or Key.Net [15]. The keypoint detection provides a
triplet $(x,y,\text{scale})$ that defines a square or circular patch. The
patch orientation is then estimated using handcrafted methods like dominant
gradient orientation [56] or center of mass [79] or learned ones like [98, 62,
55]. Optionally, the affine-covariant shape [16, 62] is estimated. Finally,
the patch is geometrically rectified and described using local patch
descriptors such as SIFT [56], HardNet [61], SOSNet [90], and others.

Recent advances in deep learning have led to the development of feature
detection and description methods that do not rely on patch extraction.
Methods like SuperPoint [30], R2D2 [76], D2Net [33] and DISK [93] employ
feedforward Convolutional Neural Networks (CNNs) and assume up-is-up image
orientation. Some recent methods have proposed learning matching directly,
such as SuperGlue [80], while others skip the detection step entirely [88, 96,
22].

Robust Estimation. Feature matching often leads to a large number of outliers
that are inconsistent with the scene geometry. This holds especially in wide-
baseline cases, where the inlier ratio often falls below 10%. Robust
estimation is thus crucial to simultaneously find the sought model (_e.g_.,
relative pose) and the matches consistent with it (its inliers). Classical
approaches employ a RANSAC-like [36] hypothesize-and-verify strategy,
iteratively applying minimal solvers [36, 45, 44, 87, 52, 53] to random
subsets of the input data until an all-inlier sample is found. To improve upon
RANSAC, various techniques have been developed, such as local optimization
methods (LO-RANSAC, LO+-RANSAC, and GC-RANSAC) [26, 54, 9], advanced scoring
functions (MLESAC, MSAC, MAGSAC, and MAGSAC++) [92, 10, 12, 4], speed-ups
using probabilistic sampling (PROSAC, NAPSAC, and P-NAPSAC) [24, 67, 12],
preemptive verification strategies (SPRT and SP-RANSAC) [25, 14], degeneracy
checks (DEGENSAC, QDEGSAC, and NeFSAC) [27, 37, 21], and methods for auto-
tuning of the inlier threshold (MINPRAN and a contrario RANSAC) [86, 63, 77].

In recent years, several neural network-based algorithms have been proposed
aiming at robust relative pose estimation. Context normalization networks [97]
is the first paper on the topic proposing to use PointNet (MLP) with batch
normalization [46] as a context mechanism. Attentive context normalization
networks [89] introduces a special architectural block for the task. Deep
Fundamental matrix estimation [74] uses differentiable iteratively re-weighted
least-squares with predicted weights. The OANet algorithm [99] introduces
several architectural blocks for correspondence filtering. Neural Guided
RANSAC [18] uses a CNe-like architecture with a different training objective.
A guided sampling algorithm exploits the predicted correspondence scores
inside RANSAC to find accurate models early. CLNet [100] introduces several
algorithmic and architectural improvements to remove gross outliers with
iterative pruning. These techniques provide alternatives for tentative
correspondence pre-filtering and weighting after the matches are formed. A
final least-squares fitting or RANSAC is then applied to obtain the model
parameters from the kept matches.

Motivation and Contributions. Jointly performing feature matching and robust
estimation is a problem of high complexity, making it impractical in the
general case. For example, when matching $n$ features in each image, the
matching complexity is $n^{2}$. Injecting this into the complexity of robust
estimation, we get $\binom{n^{2}}{m}$, where $m$ is the sample size to fit a
minimal model, such as $m=5$ for essential matrix estimation. This makes the
probability of selecting an all-inlier sample that leads to a good model
extremely low. When having $1000$ features in each image and estimating an
essential matrix, more than $10^{26}$ combinations must be tried.

We propose a new method, AffineGlue, to perform joint feature matching and
robust estimation by employing single-point solvers [83, 5, 40, 42, 34, 41].
This approach reduces the complexity of the joint procedure to that of the
matching $\mathcal{O}(n^{2})$, as $m=1$ in this special case. We use minimal
solvers that estimate the two-view geometry from a single affine
correspondence (AC) – a feature that contains higher-order information about
the underlying scene geometry [5, 6, 34]. Also, we propose a new one for
estimating the homography from a single AC. AffineGlue uses any off-the-shelf
feature matcher to form one-to-many correspondences that are finalized when
performing robust estimation and guided matching. Additionally, we train a
neural network [21] to efficiently reject ACs likely to be inconsistent with
the sought model. The proposed method outperforms state-of-the-art feature
detectors and matchers by a significant margin on a variety of real-world and
large-scale datasets.

##  2 Theoretical Background

Affine correspondence $(\mathbf{p}_{1},\mathbf{p}_{2},\mathbf{A})$ is a
triplet, where $\mathbf{p}_{1}=[u_{1}\;v_{1}\;1]^{\text{T}}$ and
$\mathbf{p}_{2}=[u_{2}\;v_{2}\;1]^{\text{T}}$ are a corresponding homogeneous
point pair in two images and $\mathbf{A}$ is a $2\times 2$ linear
transformation which is called local affine transformation. For $\mathbf{A}$,
we use the definition provided in [64] as it is given as the first-order
Taylor-approximation of the $\text{3D}\to\text{2D}$ projection function.

Fundamental matrix $(\mathbf{F})$ is a $3\times 3$ rank-$2$ matrix relating
the corresponding points $\mathbf{p}_{1}$, $\mathbf{p}_{2}$ as:

$\mathbf{p}_{2}^{\text{T}}\mathbf{F}\mathbf{p}_{1}=0.$ (1)

Essential matrix $(\mathbf{E}\in\mathbb{R}^{3\times 3})$ is related to
$\mathbf{F}$ as
$\mathbf{K}^{\prime-\text{T}}\mathbf{E}\mathbf{K}^{-1}=\mathbf{F}$, where
$\mathbf{K}$, $\mathbf{K}^{\prime}$ are the intrinsic parameters of the
cameras [44]. (1) can be written as
$\mathbf{p}_{2}^{\text{T}}\mathbf{K}^{\prime-\text{T}}\mathbf{E}\mathbf{K}^{-1}\mathbf{p}_{1}=0$.
In the rest of the paper, we assume that the corresponding points
$\mathbf{p}_{1}$, $\mathbf{p}_{2}$ have been premultiplied by matrices
$\mathbf{K}$, $\mathbf{K}^{\prime}$. This simplifies (1) to

$\mathbf{p}_{2}^{\text{T}}\mathbf{E}\mathbf{p}_{1}=0.$ (2)

Essential matrix $\mathbf{E}$ is decomposed as
$\mathbf{E}=[\mathbf{t}]_{\times}\mathbf{R}$, where
$\mathbf{R}\in\text{SO}(3)$, $\mathbf{t}\in\mathbb{R}^{3}$ is the relative
pose of the two views.

The relationship of an affine correspondence (AC) and essential matrix
$\mathbf{E}$ was first defined in [7] as

$\mathbf{A}^{-\text{T}}\mathbf{n}_{1}=-\mathbf{n}_{2},$ (3)

where $\mathbf{n}_{1}$, $\mathbf{n}_{2}$ are the normals to the epipolar lines
in the images. This linear constraint is built on two properties of ACs.
First, due to $\mathbf{A}$ being a linear approximation of the imaging
function, it transforms the infinitesimal neighborhood of $\mathbf{p}_{1}$ to
that of $\mathbf{p}_{2}$. Therefore, $\mathbf{A}$ maps the lines passing
through $\mathbf{p}_{1}$. Thus,
$\mathbf{A}\mathbf{p}_{1}\parallel\mathbf{p}_{2}$ which can be written as
$\mathbf{A}^{-\text{T}}\mathbf{n}_{1}=\beta\mathbf{n}_{2}$, where
$\mathbf{n}_{1}$, $\mathbf{n}_{2}$ are the normals to the epipolar lines and
operator $\parallel$ denotes two parallel vectors; $\beta\in\mathbb{R}$. These
normals are calculated as the first two coordinates of the epipolar lines as
$\mathbf{n}_{1}=\mathbf{l}_{1[1:2]}=(\mathbf{E}^{\text{T}}\mathbf{p}_{2})_{[1:2]}$,
$\mathbf{n}_{2}=\mathbf{l}_{2[1:2]}=(\mathbf{E}\mathbf{p}_{1})_{[1:2]}$. Since
$\mathbf{n}_{1}$ and $\mathbf{n}_{2}$ absorb the scaling from $\mathbf{E}$,
scalar $\beta$ is $-1$. In summary, an affine correspondence imposes three
independent constraints on the essential matrix. One is given by (2), and two
others by (3).

##  3 Joint Matching and Estimation

A method is proposed in this section to robustly estimates the parameters of
the sought model while simultaneously performing feature matching. See Fig. 1.
The pseudo-code of the algorithm is as follows:

$\mathcal{P}_{1}$, $\mathcal{P}_{2}$ – data points in the two images

$\mathcal{M}^{*}$ – correspondences, $\theta$ – model params.

$\theta^{*}\leftarrow\textbf{0},q^{*}\leftarrow 0$,
$\mathcal{M}^{*}\leftarrow\varnothing$ $\triangleright$ Initialization

while $\neg$Terminate() do

$\mathcal{S}\leftarrow\text{NextBestMatch}(\mathcal{P}_{1},\mathcal{P}_{2})$
$\triangleright$ Generate a match

$\theta\leftarrow\text{EstimateModel}(\mathcal{S})$ $\triangleright$ A one-
point solver

$\mathcal{M}\leftarrow\text{GuidedMatching}(\theta,\mathcal{P}_{1},\mathcal{P}_{2})$

$q\leftarrow\text{GetScore}(\theta,\mathcal{M})$

if $q>q^{*}$ then $\triangleright$ Update the best model

$q^{\prime},\theta^{\prime},\mathcal{M}^{\prime}\leftarrow\text{LocalOptimization}(\theta,\mathcal{P}_{1},\mathcal{P}_{2})$

$\theta^{*}\leftarrow\theta^{\prime},q^{*}\leftarrow q^{\prime}$,
$\mathcal{M}^{*}\leftarrow\mathcal{M}^{\prime}$

Similar to RANSAC, we formalize the problem as iterative sampling, model
estimation, and scoring. We assume, however, to have a minimal solver that
estimates the model parameters from a single match. This allows formalizing
function NextBestMatch that forms sample $\mathcal{S}$ consisting of a single
correspondence ($\textbf{p}_{1}$, $\textbf{p}_{2}$, $\mathbf{A}$) in each
iteration, where $\textbf{p}_{1}\in\mathcal{P}_{1}$ and
$\textbf{p}_{2}\in\mathcal{P}_{2}$ are points in the images, and
$\mathbf{A}\in\mathbb{R}^{2\times 2}$ is the local affine frame. Model
$\theta$ is estimated from $\mathcal{S}$. Note that the method works with any
single-point solver, _e.g_. [83], not only with ones leveraging ACs.

After estimating the model, we perform guided matching [85, 58, 11] using
model $\theta$ to find a set $\mathcal{M}$ of correspondences consistent with
the model parameters. The model quality $q$ is calculated from $\mathcal{M}$,
_e.g_., as its support (_i.e_., $|\mathcal{M}|$), or by any existing scoring
technique. In case a new best model is found, we apply local optimization to
improve its accuracy. The algorithm runs until the termination criterion is
triggered. Next, we will describe each step in depth.

Next Best Match Selection. Suppose that we are given
$n_{1},n_{2}\in\mathbb{N}^{+}$ features in the first and second images,
respectively. Forming correspondences is of quadratic complexity
$\mathcal{O}(n_{1}n_{2})$. Thus, iterating through all possible matches, while
doable, severely affects the run-time. To alleviate this computational burden,
we obtain the $k$ best matches for each feature in the source image, where
$k\ll n_{2}$, $k\in\mathbb{N}^{+}$. This can be done by applying the standard
$k$-nearest-neighbors ($k$NN) descriptor matching. Algorithms like SuperGlue,
solving the optimal transport problem, provide a score matrix via the Sinkhorn
algorithm [51]. In this case, the $k$ best matches are the $k$ features with
the highest scores. This allows AffineGlue to explore the $k$ best matches and
thus, reduce the matching ambiguity – for example, see Fig. 1, where the
potential matches are on the windows, and existing matchers have a hard time
finding the correct correspondence.

Still, the probability of finding a good match when uniformly randomly
sampling from $kn_{1}$ correspondences can be low in practice, leading to many
iterations and high runtimes. Thus, we follow a PROSAC-like [24] procedure
where the potential matches are ordered by a quality prior. First, we select
the correspondence that is the most likely to be correct, and then,
progressively, we sample from less likely ones. This prior either comes
directly from the applied matcher or is predicted by a deep network. In this
paper, we train the recent NeFSAC [21] to predict the probability of each AC
leading to an accurate model. The exact procedure is described in the supp.
material.

Scoring and Guided Matching. Assume that we are given a model
$\theta\in\mathbb{R}^{d_{\theta}}$ estimated from a single correspondence
($d_{\theta}\in\mathbb{N}$ is the dimensionality of the model manifold), point
sets $\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ in the two images, and a point-
to-model residual function
$\phi:\mathbb{R}^{d_{\theta}}\times\mathbb{R}^{d_{p}}\to\mathbb{R}$, where
$d_{p}\in\mathbb{N}$ is the data dimension. Model $\theta$ can be, for
example, an essential matrix and $\phi$ the Sampson distance or symmetric
epipolar error. In short, we iterate through all potential matches; select the
pair with the lowest point-to-model residual for each point in the first
image; and, finally, calculate the score from the selected correspondences.
The pseudo-code for the guided sampling is as follows:

$\mathcal{P}_{1}$ \- points, $\theta$ \- model, $H$ \- hashing fn.

$K$ \- $k$ best match, $\epsilon$ \- thr., $W$ \- weight fn., $Q$ \- scoring

$\mathcal{M}$ \- correspondences, $q$ \- model score

$\mathcal{M}\leftarrow\varnothing$ $\triangleright$ Initialization to empty
set

for each $\textbf{p}_{1}\in\mathcal{P}_{1}$ do $\triangleright$ Each point in
the 1st image

$r^{*}\leftarrow\epsilon$, $\textbf{p}_{2}^{*}\leftarrow\textbf{0}$
$\triangleright$ Best residual and match

for each $\textbf{p}_{2}\in(K(\textbf{p}_{1})\cap H(\textbf{p}_{1},\theta))$
do

if $\phi((\textbf{p}_{1},\textbf{p}_{2}),\theta)<r^{*}$ then

$r^{*}\leftarrow\phi((\textbf{p}_{1},\textbf{p}_{2}),\theta)$,
$\textbf{p}_{2}^{*}\leftarrow\textbf{p}_{2}$

if $r^{*}<\epsilon$ then

$\mathcal{M}\leftarrow\mathcal{M}\cup\\{(\textbf{p}_{1},\textbf{p}_{2}^{*})\\}$

$q\leftarrow q+W(K(\textbf{p}_{1}))Q(\theta)$

The inputs of the algorithm are the points in the first image
$\mathcal{P}_{1}$; model $\theta$; a function
$K:\mathcal{P}_{1}\to\mathcal{P}_{2}^{k}$ assigning the $k$ best match in the
second image to a point in the first one; the inlier-outlier threshold
$\epsilon\in\mathbb{R}^{+}$; a weighting $W:\mathbb{R}\to\mathbb{R}$, a model
scoring $Q:\mathbb{R}^{d}\to\mathbb{R}$, and a hashing function
$H:\mathcal{P}_{1}\times\mathbb{R}^{d}\to\mathcal{P}_{2}^{*}$. We use MAGSAC++
[12] as $Q$ to calculate the model score via marginalizing over an acceptable
range of noise scale $\sigma$.

Given point $\textbf{p}_{1}$ and model $\theta$, the purpose of the hashing
function $H$ is to efficiently select matches from $\mathcal{P}_{2}$ that are
consistent with $\theta$ when paired $\textbf{p}_{1}$, _i.e_.,
$\forall\textbf{p}_{2}\in
H(\textbf{p}_{1},\theta):\phi(\textbf{p}_{1},\textbf{p}_{2})\leq\epsilon$.
Such $H$ can be easily constructed for homographies or rigid transformations
using regular grids. Also, one can use epipolar hashing [9] when estimating
relative pose. In cases, where no such function exists for a particular model,
$H$ can be omitted without affecting the accuracy.

We found that it is important to use a weighting $W$ in the score calculation,
especially, when estimating relative pose, _i.e_., fundamental or essential
matrix. The reason is that the point-to-model residual (_e.g_., Sampson
distance) being zero, does not necessarily mean that it is a correct
correspondence. We are not able to measure the translation along the epipolar
lines. Without accounting for this, the procedure tends to hallucinate a large
amount of incorrect matches that are consistent with the found model. The
model has lots of inliers, while being incorrect. Therefore, for cases with
such residual functions, we introduce an additional parameter $\mu\in[0,1]$
that will act similarly as the Lowe ratio threshold [56] or Wald criterion
[95]. For each point $\textbf{p}_{1}$, we are given
$K(\textbf{p}_{1})=\\{\textbf{p}_{2}^{1},...,\textbf{p}_{2}^{k}\\}$ with
matching scores $S(\textbf{p}_{1})=\\{s_{12}^{1},...s_{12}^{k}\\}$ from the
feature matcher. We only keep those potential matches from
$K(\textbf{p}_{1})$, where the matching score $s_{12}^{i}\geq\mu\left(\max
S(\textbf{p}_{1})\right)$. Thus,
$K^{\prime}(\textbf{p}_{1})=\\{\textbf{p}_{2}^{i}\;|\;\textbf{p}_{2}^{i}\in
K(\textbf{p}_{1})\wedge s_{12}^{i}\geq\mu\left(\max
S(\textbf{p}_{1})\right)\\}$ Weight
$W(\textbf{p}_{1})=|K^{\prime}(\textbf{p}_{1})|^{-1}$ in the proposed
algorithm. Thus, the weight is inversely proportional to the number of matches
that have similar matching scores.

Local Optimization. As it was discussed in [13, 8], inner RANSAC-based local
optimization is crucial when using ACs. Thus, when a new best model is found,
we apply a few iterations of RANSAC on the selected matches using a point-
based solver, ignoring the affine shapes. For example, this means that the
refitting is done by the 5PC [87] algorithm when estimating E matrices. In
practice, the LO runs only $\log t$ times [26], where $t$ is the total
iteration number of the outer loop. The iteration number spent inside the
local optimization is typically set to a small value, _e.g_., $20$.

##  4 Homography from 1AC

In this section, we propose a new minimal solver for homography estimation
using a single affine correspondence as input, and assume the gravity
direction to be known. While requiring the gravity might seem a restrictive
constraint, assuming that it points downwards and is $[0,-1,0]^{\text{T}}$ is
a reasonably good assumption in practice and it works in all our experiments.

Homography matrix $\mathbf{H}\in\mathbb{R}^{3}$ is defined as
$\mathbf{H}=\mathbf{R}-\frac{1}{d}\mathbf{t}\mathbf{n}^{\text{T}}$, where
$\mathbf{R}\in\text{SO}(3)$ and $\mathbf{t}\in\mathbb{R}^{3}$ are the relative
camera rotation and translation, respectively, $d\in\mathbb{R}$ is the plane
intercept and $\mathbf{n}\in\mathbb{R}^{3}$ is its normal. To solve for
$\mathbf{H}$, first, we derive the constraints for relative pose
$\mathbf{R},\mathbf{t}$ from a single AC
$(\mathbf{p}_{1},\mathbf{p}_{2},\mathbf{A})$, and the vertical directions
$\mathbf{v}_{1}=[x_{v_{1}},y_{v_{1}},z_{v_{1}}]^{\text{T}},\mathbf{v}_{2}=[x_{v_{2}},y_{v_{2}},z_{v_{2}}]^{\text{T}}$
known in both images. The relative pose with a known vertical direction has
three degrees-of-freedom (DoF), and the AC imposes three constraints on it.

---
Figure 2: Stability study. The frequencies ($100$k runs) of $\text{log}_{10}$
rotation and translation errors (both in degrees) in the homography estimated
by the 4PC [44], 2AC [5], and proposed 1AC+G(H) solvers.

According to [49], we can express the rotation matrix as
$\mathbf{R}=\mathbf{R}_{2}^{\text{T}}\mathbf{R}_{y}\mathbf{R}_{1}$, where
$\mathbf{R}_{y}$ is a rotation around $y$-axis, $\mathbf{R}_{1}$ transforms
$\mathbf{v}_{1}$ to $y$-axis, $\mathbf{R}_{2}$ transforms $\mathbf{v}_{2}$ to
$y$-axis. Let $\mathbf{y}=[0,\ 1,\ 0]^{\text{T}}$ be the $y$-axis. The axis of
$\mathbf{R}_{1}$ can be computed as
$\mathbf{v}_{1}\times\mathbf{y}=[-z_{v_{1}}/d,\ \ 0,\ \
x_{v_{1}}/d]^{\text{T}}$, where $d=x_{v_{1}}^{2}+z_{v_{1}}^{2}$, the angle is
obtained as
$\arccos{(\mathbf{v}_{1}^{\text{T}}\mathbf{y})}=\arccos{(y_{v_{1}})}$.
Rotation matrix $\mathbf{R}_{1}$ is computed using the Rodrigues formula,
rotation matrix $\mathbf{R}_{2}$ is obtained similarly. Matrix
$\mathbf{R}_{y}$ is expressed elementwise as

$\mathbf{R}_{y}=\frac{1}{1+x^{2}}\begin{bmatrix}1-x^{2}&0&-2x\\\
0&1+x^{2}&0\\\ 2x&0&1-x^{2}\end{bmatrix},$ (4)

where $x=\tan\phi/2$. Now, we can express the essential matrix $\mathbf{E}$ as
$\mathbf{E}=\mathbf{R}_{2}^{\text{T}}[\mathbf{t}^{\prime}]_{\times}\mathbf{R}_{y}\mathbf{R}_{1}$,
where $\mathbf{t}^{\prime}=\mathbf{R}_{2}\mathbf{t}$.

Let $\mathbf{q}_{1}=\mathbf{R}_{1}\mathbf{p}_{1}$ and
$\mathbf{q}_{2}=\mathbf{R}_{2}\mathbf{p}_{2}$. Eq. (2) becomes

$\mathbf{q}_{2}^{\text{T}}[\mathbf{t}^{\prime}]_{\times}\mathbf{R}_{y}\mathbf{q}_{1}=0.$
(5)

In order to modify constraints (3) in a similar way, we first define
$\mathbf{B}=\mathbf{A}^{-\text{T}}[\mathbf{r}_{1}^{1}\,\mathbf{r}_{1}^{2}]^{\text{T}}$,
$\mathbf{C}=[\mathbf{r}_{2}^{1}\,\mathbf{r}_{2}^{2}]^{\text{T}}$, where
$\mathbf{r}_{i}^{1}$, $\mathbf{r}_{i}^{2}$ $\mathbf{r}_{i}^{3}$ are the column
vectors of $\mathbf{R}_{i}$, $i\in\\{1,2\\}$.

The elements of $\mathbf{B}$ are written in row-major order as
$b_{1},...,b_{6}$, and the elements of $\mathbf{C}$ as $c_{1},...,c_{6}$. We
can rewrite the constraints (3) as

$\small\begin{split}\mathbf{A}^{-\text{T}}\mathbf{n}_{1}-\mathbf{n}_{2}=\mathbf{A}^{-\text{T}}\mathbf{l}_{1[1:2]}-\mathbf{l}_{2[1:2]}\\\
=\mathbf{A}^{-\text{T}}[\mathbf{r}_{1}^{1}\
\mathbf{r}_{1}^{2}]^{\text{T}}\mathbf{R}_{y}^{\text{T}}[\mathbf{t}^{\prime}]_{\times}^{\text{T}}\mathbf{q}_{2}-[\mathbf{r}_{2}^{1}\
\mathbf{r}_{2}^{2}]^{\text{T}}[\mathbf{t}^{\prime}]_{\times}\mathbf{R}_{y}\mathbf{q}_{1}=0.\end{split}$
(6)

Constraints (5), (6) give $3$ equations in variables $x\in\mathbb{R}$ and
$\mathbf{t}^{\prime}\in\mathbb{R}^{3}$. After multiplying the equations with
$1+x^{2}$, we get three equations that are linear in the elements of
translation $\mathbf{t}^{\prime}$. We can, therefore, use the hidden variable
approach to rewrite the equations in the form
$\mathbf{M}(x)\mathbf{t}^{\prime}=0$, where $\mathbf{M}(x)$ is a $3\times 3$
matrix whose elements depend on $x$. If $(x,\mathbf{t}^{\prime})$ is a
solution to the linear system, then matrix $\mathbf{M}(x)$ must be singular.
Consequently, $\det\mathbf{M}(x)=0$ holds. This is a univariate polynomial of
degree $6$. We find its roots as the eigenvalues of its companion matrix.
After finding $x$, we calculate $\mathbf{t}^{\prime}$ as the kernel of matrix
$\mathbf{M}(x)$ and the rotation $\mathbf{R}_{y}$ according to (4). Finally,
we compute the relative pose $(\mathbf{R},\mathbf{t})$ as
$\mathbf{R}=\mathbf{R}_{2}^{\text{T}}\mathbf{R}_{y}\mathbf{R}_{1}$,
$\mathbf{t}=\mathbf{R}_{2}^{\text{T}}\mathbf{t}^{\prime}$.

Next, we will solve for the unknown plane parameters using the estimated
relative pose. We can set $\mathbf{n}^{\prime}=\frac{1}{d}\mathbf{n}$ and
simplify the expression as follows:

$\mathbf{H}=\mathbf{R}-\mathbf{t}\mathbf{n}^{\prime\text{T}}.$ (7)

To find the homography $\mathbf{H}$ consistent with both the affine
correspondence $(\mathbf{p}_{1},\mathbf{p}_{2},\mathbf{A})$ and vertical
directions $\mathbf{v}_{1}$ and $\mathbf{v}_{2}$, we substitute
$(\mathbf{R},\mathbf{t})$ into (7). Then, we only need to find vector
$\mathbf{n}^{\prime}\in\mathbb{R}^{3}$. We substitute the expression (7) into
the constraints from [7] connecting affine correspondences and homography
$\mathbf{H}$. We obtain $6$ linear equations in $3$ unknowns. They are shown
in the supp. mat. The LS method obtains vector $\mathbf{n}^{\prime}$ from the
above system. Finally, we compute the homography $\mathbf{H}$ from
$\mathbf{R}$, $\mathbf{t}$, $\mathbf{n}^{\prime}$ using the equation (7).

##  5 Experiments

This section first tests the proposed minimal solver in a fully controlled
synthetic environment. Then AffineGlue is evaluated on real-world datasets for
relative pose and homography estimation. All experiments were implemented in
C++ and performed on an Intel(R) Core(TM) i9-10900K CPU @ 3.70GHz.

Synthetic Experiments. To create a synthetic scene, we generate two cameras
with random rotations and translations and focal length set to $1000$. A
randomly oriented 3D point is generated and projected into both cameras. The
affine transformation is calculated from the point orientation. We generated
$100$k random problem instances and ran the solvers on noiseless samples. Fig.
2 shows histograms of the $\log_{10}$ rotation and translation errors. The
plots show that all solvers are stable – there is no peak close to $10^{0}$.
In Fig. 3, the average errors in degrees are shown as a function of the image
noise. We use a fixed gravity (0.1∘) and affine noise ($0.5$ px). It is
important to note that the realistic affine noise is unclear in practice, with
no work analyzing it. These plots only intend to demonstrate that the solvers
act reasonably w.r.t. increasing noise levels, which they do. More synthetic
experiments are in the supplementary material.

|
---|---
Figure 3: Image noise study. The average (over $100$k runs) angular errors of
the rotations and translation estimated by the 4PC [44], 2AC [5], and proposed
1AC+G(H) homography solvers plotted as a function of the image noise in
pixels.

Features | Estimator | Solver |  AVG $\downarrow$ |  MED $\downarrow$ |  AUC@$1^{\circ}\uparrow$ |  @$2.5^{\circ}\uparrow$ |  @$5^{\circ}\uparrow$ |  @$10^{\circ}\uparrow$ |  @$20^{\circ}\uparrow$ | # inliers
---|---|---|---|---|---|---|---|---|---|---
SuperPoint + SuperGlue | AffineGlue |  1AC+$i$G |  12.6 | 0.7 | 34.5 | 55.9 | 70.3 | 81.3 | 89.2 | 394
1AC+mD |  12.6 | 0.8 | 34.5 | 56.0 | 70.4 | 81.4 | 89.2 | 395
MAGSAC++ | 5PC |  14.1 | 1.3 | 23.0 | 43.5 | 59.9 | 74.1 | 84.6 | 276
1PC+$i$G |  14.0 | 1.3 | 23.0 | 43.4 | 59.6 | 74.0 | 84.7 | 276
DoG-8k + HardNet + AffNet | AffineGlue |  1AC+$i$G |  13.4 | 0.7 | 38.7 | 57.4 | 70.0 | 79.9 | 87.4 | 286
1AC+mD |  15.2 | 0.9 | 22.2 | 50.6 | 62.6 | 73.0 | 81.7 | 202
MAGSAC++ | 5PC |  16.3 | 1.4 | 27.7 | 42.7 | 54.3 | 66.2 | 77.2 | 210
1AC+$i$G |  15.1 | 0.9 | 33.3 | 50.5 | 62.5 | 72.9 | 81.6 | 257
DoG-8k + HardNet + Adalam |  MAGSAC++ | 5PC |  18.8 | 0.8 | 34.3 | 52.5 | 65.0 | 74.8 | 82.4 | 307
LoFTR | 5PC |  13.6 | 1.3 | 22.5 | 43.4 | 59.6 | 73.7 | 84.5 | 866
LoFTR |  3PC+$i$G |  14.1 | 1.4 | 21.0 | 40.9 | 56.7 | 71.1 | 82.6 | 878
DISK | 5PC |  14.7 | 0.9 | 27.9 | 44.3 | 55.7 | 64.5 | 71.2 | 474
DISK |  3PC+$i$G |  14.5 | 0.8 | 29.1 | 45.8 | 57.1 | 66.1 | 72.9 | 617
R2D2 + NN | 5PC | 13.0 | 2.7 | 13.6 | 28.8 | 42.9 | 57.9 | 70.3 | 169
R2D2 + NN |  3PC+$i$G | 12.9 | 2.7 | 13.9 | 28.8 | 42.8 | 57.5 | 70.2 | 169
DoG-8k + SOSNet + NN | 5PC | 40.4 | 5.9 | 12.8 | 23.9 | 33.5 | 43.3 | 52.9 |  155
DoG-8k + SOSNet + NN |  3PC+$i$G | 40.4 | 5.9 | 12.9 | 23.8 | 33.4 | 43.3 | 52.9 |  155

Table 1:  Relative pose estimation on PhotoTourism [48] on a total of 9900
image pairs. We report the avg. and median pose errors (in degrees; max. of
the translation and rotation errors), their AUC scores and the inlier numbers.
We use the 3PC+iG [32] and the 1AC+iG [40] solvers with identity gravity, the
1AC+mD solver [34] on depth from MiDaS-v3 [73, 75], and the five point method
(5PC) [68]. For solvers requiring more than a single match, we apply the
state-of-the-art MAGSAC++ [12]. Finally, the Levenberg-Marquardt method [65]
minimizes the pose error on all inliers. The best values are bold in each
group. The absolute best ones are underlined.

###  5.1 Real-World Experiments

Features | Estimator | Solver |  AVG $\downarrow$ |  MED $\downarrow$ |  AUC@$1^{\circ}\uparrow$ |  @$2.5^{\circ}\uparrow$ |  @$5^{\circ}\uparrow$ |  @$10^{\circ}\uparrow$ |  @$20^{\circ}\uparrow$ | # inliers
---|---|---|---|---|---|---|---|---|---|---
SuperPoint + SuperGlue | AffineGlue |  1AC+$i$G | 12.9 |  15.8 | 0.8 | 7.1 | 20.6 | 39.7 | 58.4 | 119
1AC+mD | 14.0 |  15.5 | 0.8 | 7.0 | 20.7 | 39.8 | 58.1 | 110
MAGSAC++ | 5PC | 21.4 |  16.5 | 0.7 | 5.9 | 17.3 | 33.9 | 50.9 |  189
3PC+iG | 32.4 | 21.0 | 0.5 | 4.2 | 11.5 | 21.9 | 33.1 |  184
DoG-8k + HardNet + AffNet | AffineGlue |  1AC+$i$G | 26.8 | 15.0 | 0.7 | 5.0 | 13.0 | 24.2 | 37.2 | 146
1AC+mD | 24.7 | 12.4 | 0.6 | 4.5 | 12.6 | 25.3 | 39.6 | 120
MAGSAC++ | 5PC | 33.7 | 29.9 | 0.3 | 2.3 |  16.6 | 13.6 | 22.9 |  181
1AC+iG | 25.3 | 13.0 | 0.3 | 3.1 |  19.0 | 18.4 | 29.4 |  164
DoG-8k + HardNet + Adalam |  MAGSAC++ | 5PC | 54.1 | 17.8 | 0.5 | 3.7 | 11.1 | 22.3 | 34.9 | 101
LoFTR | 5PC | 30.3 |  16.6 | 1.1 | 8.3 | 22.5 | 41.2 | 57.7 | 468
R2D2 + NN | 5PC | 32.9 | 13.6 | 0.6 | 4.2 | 12.0 | 24.6 | 38.1 | 190
R2D2 + NN |  3PC+iG | 18.9 | 10.6 | 0.4 | 2.8 |  18.2 | 16.8 | 27.4 | 137
DoG-8k + SOSNet + NN | 5PC | 33.3 | 29.7 | 0.4 | 2.6 |  16.6 | 13.6 | 23.4 |  178
DoG-8k + SOSNet + NN |  3PC+iG | 60.8 | 36.4 | 0.3 | 1.6 |  15.3 | 12.4 | 22.5 |  138

Table 2:  Relative pose estimation on ScanNet [28] on the 1500 image pairs
from [80, 88]. We report the avg. and median pose errors (in degrees; max. of
the translation and rotation errors), their AUC scores and the inlier numbers.
We use the 3PC+iG [32] and 1AC+iG [40] solvers with identity gravity, the
1AC+mD solver [34] on depth from MiDaS-v3 [73, 75], and the five point method
(5PC) [68]. For solvers requiring more than a single match, we apply the
state-of-the-art MAGSAC++ [12]. Finally, the Levenberg-Marquardt method [65]
minimizes the pose error on all inliers. The best values are bold in each
group. The absolute best ones are underlined.

Features | Estimator | Solver |  AUC@$1\text{px}\uparrow$ |  @$2.5\text{px}\uparrow$ |  @$5\text{px}\uparrow$ |  @$10\text{px}\uparrow$ |  Time (secs) $\downarrow$
---|---|---|---|---|---|---|---
SuperPoint + SuperGlue | AffineGlue |  1AC+$i$G-H | 50.5 | 73.9 | 84.9 | 91.1 | 0.04
MAGSAC++ |  1AC+$i$G-H | 45.6 | 71.7 | 83.9 | 90.9 | 0.66
4PC | 37.9 | 65.6 | 79.0 | 90.1 | 0.60
DoG-2k + HardNet + AffNet | AffineGlue |  1AC+$i$G-H | 40.1 | 68.0 | 81.4 | 88.8 | 0.29
MAGSAC++ |  1AC+$i$G-H | 40.3 | 68.8 | 82.3 | 89.8 | 0.11
4PC | 40.9 | 69.3 | 82.7 | 90.4 | 0.01
LoFTR |  MAGSAC++ | 4PC | 41.8 | 68.6 | 81.2 | 87.9 | 0.40
DoG-2k + SOSNet + NN |  1AC+$i$G-H | 38.3 | 65.5 | 79.5 | 87.4 | 0.47
DoG-2k + SOSNet + NN | 4PC | 36.9 | 63.3 | 77.0 | 85.1 | 0.25
R2D2 + NN |  1AC+$i$G-H | 27.6 | 51.5 | 65.9 | 75.1 | 0.20
R2D2 + NN | 4PC | 27.4 | 51.0 | 65.5 | 75.4 | 0.09
DISK + NN |  1AC+$i$G-H | 25.1 | 51.8 | 68.5 | 77.8 | 0.29
DISK + NN | 4PC | 25.0 | 51.5 | 68.1 | 78.7 | 0.20

Table 3:  Homography estimation on the HPatches dataset [3]. The AUC scores
and avg. times are reported. AffineGlue is applied with the proposed 1AC+iG-H
solver assuming identity gravity. We also run MAGSAC++ [12] with the 4PC [44]
and 1AC+iG-H solvers. The best values are bold in each group, the absolute
bests are underlined.

Affine Features. There are multiple ways to obtain affine features from real
images. First, the most standard is to use a local feature detector, like DoG
[56] or Key.Net [15], estimate keypoint locations and scales, and use the
patch-based AffNet [62] to get affine shapes. Finally, a patch-based
descriptor, like HardNet [61] or SOSNet [90], is applied. This approach is
among leaders in IMC 2020 benchmark [48].

The second way is to use handcrafted affine detectors, such as MSER [59] and
W$\alpha$SH [94], that jointly estimate local feature geometry including
affine shape. On top of these features, we can detect any patch-based
descriptors, _e.g_., HardNet [61] or SOSNet [90].

Finally, we experimented with joint detector-descriptor models, such as
SuperPoint [30] and DISK [93], which outputs keypoint location and descriptor.
To upgrade point-features to affine-features, we employ Self-Scale-Ori [55]
scale estimator to get the scale and orientation. Finally, AffNet runs to get
affine shape. Note, it gives a user 2 options – either use original
SuperPoint/DISK descriptors or patch-based HardNet on top of affine feature.

In the main experiments, we run the proposed AffineGlue on DoG + HardNet +
AffNet + NN (NN – nearest neighbor matching) and SuperPoint + Self-Scale-Ori +
AffNet + SuperGlue features since they lead to the most accurate results –
this will be shown in the ablation study. Obtaining a pool of potential
matches is straightforward when using NN on HardNet descriptors. To get a
similar pool for SuperGlue, we directly access the matching score matrix that
is obtained when solving the optimal transport problem. This allows selecting
the $k$ best matches for each point.

Minimal Solvers. When testing relative pose estimation, we compare three
solvers. 5PC [87] is the widely-used algorithm estimating the pose from five
point correspondences. The 1AC+mD solver is proposed in [34]. It estimates the
pose from a single AC and predicted monocular depth. To allow running this
solver, we obtain relative depth by MiDaS-v3 [73, 75]. We also compare solver
1AC+G [40] that requires a single AC and a known direction in the images. To
demonstrate the robustness of the proposed AffineGlue, we always run 1AC+G
assuming that the gravity points downwards – its direction is
$[0,-1,0]^{\text{T}}$. Thus, we call the solver 1AC+$i$G. This way, we do not
require to know the gravity direction prior to running the algorithm. This is
based on two assumptions that proved true on the tested datasets: (i) people
tend to roughly align their cameras with the gravity direction [71, 48]; (ii)
AffineGlue is robust enough so if the estimated noisy model is able to select
a few inliers, the local optimization procedure recovers. We also test the
3PC+G [32] solver that requires three PCs and the gravity. Similarly as
before, we use identity gravity.

Relative Pose – PhotoTourism. For testing the methods, we use the data from
the CVPR IMC 2020 PhotoTourism challenge [48]. It consists of 25 scenes (2 –
validation; 12 – training; 11 – test sets) of landmarks with photos of varying
sizes collected from the internet. NeFSAC is trained by splitting the training
set into two disjoint sets for training and validation. The algorithms are
tested on the two scenes for validation – a total of 9900 pairs. For robust
estimation, we chose MAGSAC++ [12] as competitor. We compare the following
detectors: SuperPoint [30] with SuperGlue [80], DoG [56] with HardNet [61]
descriptors, DoG with HardNet followed by Adalam [20], DoG with SOSNet [90]
descriptors, DISK [93], and R2D2 [76]. Also, we show the results of LoFTR
[88]. The average error of the gravity prior $[0,-1,0]^{\text{T}}$ is
$10.8^{\circ}$.

The results are in Table 1. We report the average and median pose errors
(_i.e_., the max. of the rotation and translation errors) in degrees, the AUC
scores at $1^{\circ}$, $2.5^{\circ}$, $5^{\circ}$, $10^{\circ}$, and
$20^{\circ}$, and the average inlier number. Note that the inlier number is
not informative when different detectors and matchers are compared. We show it
to highlight that the proposed method increases the inlier number compared to
MAGSAC++ with 5PC on the same features. DoG+HardNet+AffineGlue and
SP+SG+AffineGlue, on par, lead to the best results. Compared to the best
method with MAGSAC++ (_i.e_., DoG+HardNet+Adalam+5PC), DoG+HardNet+AffineGlue
improves at least 5 AUC points in all metrics. Moreover, let us highlight that
using the AC+$i$G solver instead of 5PC in MAGSAC++, improves DoG+HardNet by a
large margin, _i.e_., $5$-$8$ AUC points. Interestingly, using the 3PC+$i$G
[32] solver only marginally improves the results of MAGSAC++. There is no
significant difference in the results of the 1AC+$i$G and 1AC+$m$D solvers.
Thus, we suggest using the 1AC+$i$G as it does not require running a depth
predictor.

Relative Pose – ScanNet. The ScanNet dataset [28] contains 1613 monocular
sequences with ground truth poses and depth maps. We evaluate our method on
the 1500 pairs used in SuperGlue [80] and [88]. These pairs contain wide
baselines and extensive texture-less regions. The avg. error of the gravity
prior is $24.8^{\circ}$.

The results are shown in Table 2. We can see similar results as for
PhotoTourism. AffineGlue with DoG or SuperPoint+SuperGlue features improves
the performance by a large margin. It makes SuperPoint+SuperGlue comparable to
the detector-less LoFTR [88] with achieving even smaller avg. and med. errors
and higher AUC@$20^{\circ}$. DoG+HardNet with AffineGlue is less accurate than
SP+SG, however, it still is among the top-performing methods. Both 1AC+iG and
1AC+mD lead to similar accuracy.

Homography – HPatches. The [3] dataset contains 52 sequences under significant
illumination changes and 56 sequences that exhibit large viewpoint variation.
Since the intrinsic matrices are not provided in HPatches, we calibrate the
cameras of the 56 sequences with viewpoint changes by the RealityCapture
software [19]. We use these sequences in the evaluation.

The results are shown in Table 3. The proposed AffineGlue with
SuperPoint+SuperGlue leads to the most accurate results while being one of the
fastest algorithms. Its AUC@1∘ score is increased by $5$ AUC points compared
to the second most accurate method.

Run-time. As reported in Table 3, the avg. run-time of AffineGlue on H
estimation from SuperPoint+SuperGlue features is $0.04$ seconds. The avg. time
of pose estimation on PhotoTourism is $0.09$ and on ScanNet is $0.03$ seconds.
The avg. inference time of NeFSAC is $1.1$ ms per image pair. For comparison,
MAGSAC++ with the 5PC solver runs, on average, for $0.01$ secs on ScanNet and
for $0.04$ secs on PhotoTourism. Even though AffineGlue is slower, it still
runs in real-time while achieving state-of-the-art accuracy.

Feature Ablation. We compared a number of affine detectors to choose the best
ones. The AUC scores on PhotoTourism are shown in Table 5 and on ScanNet in
Table 4. On PhotoTourism, we used the 1AC+iG solver. On ScanNet, we used
1AC+mD. All methods use AffineGlue. DoG with HardNet and AffNet is on par with
SuperPoint with SuperGlue on PhotoTourism. On ScanNet, SP+SG is the best.
Interestingly, SuperPoint works better with HardNet descriptors than its own
when NN matching is used. As expected, classical affine shape detectors,
_i.e_. MSER and W$\alpha$SH, are inaccurate even with HardNet descriptors.

Detector | Desc. | +AffNet |  AUC@$1^{\circ}$ | $2.5^{\circ}$ | $5^{\circ}$ | $10^{\circ}$ | $20^{\circ}$
---|---|---|---|---|---|---|---
DoG-8k [56] |  HardNet+NN | ✓ | 0.5 | 4.5 | 12.6 | 25.3 | 39.6
SP [30] | ✓ | 0.4 | 2.6 |  17.7 | 16.3 | 26.9
DISK [93] | ✓ | 0.3 | 2.2 |  16.3 | 13.4 | 21.3
Key.Net [15] | ✓ | 0.3 | 1.8 |  15.3 | 10.7 | 17.4
MSER [59] | ✗ | 0.1 | 1.2 |  13.5 |  17.2 | 12.5
W$\alpha$SH [94] | ✗ | 0.0 | 0.1 |  10.5 |  11.9 |  15.7
SP [30] | +NN | ✓ | 0.6 | 4.2 | 11.7 | 23.1 | 36.1
SP [30] | +SG | ✓ | 0.8 | 7.0 | 20.7 | 39.8 | 58.1
DISK [93] | +NN | ✓ | 0.3 | 2.4 |  17.2 | 14.7 | 25.1

Table 4: Affine features on Scannet [28] used inside AffineGlue on a total of
1500 image pairs.

Detector | Desc. | +AffNet |  AUC@$1^{\circ}$ | $2.5^{\circ}$ | $5^{\circ}$ | $10^{\circ}$ | $20^{\circ}$
---|---|---|---|---|---|---|---
DoG-8k [56] |  HardNet+NN | ✓ | 38.7 | 57.4 | 70.0 | 79.9 | 87.4
Key.Net [15] | ✓ | 22.6 | 38.8 | 51.1 | 62.7 | 73.6
DISK [93] | ✓ | 16.4 | 27.7 | 37.9 | 49.6 | 63.0
MSER [59] | ✗ | 13.6 | 24.3 | 34.4 | 46.2 | 58.6
SP [30] | ✓ | 11.5 | 22.0 | 31.6 | 42.9 | 55.4
W$\alpha$SH [94] | ✗ |  10.0 |  10.1 |  10.8 |  14.0 | 13.6
SP [30] | +NN | ✓ |  18.7 | 17.5 | 26.4 | 37.0 | 48.7
SP [30] | +SG | ✓ | 34.5 | 55.9 | 70.3 | 81.3 | 89.2
DISK [93] | +NN | ✓ | 30.1 | 47.3 | 59.5 | 69.6 | 77.7

Table 5: Affine features on PhotoTourism [48] used inside AffineGlue on a
total of 9900 image pairs.

##  6 Conclusion

We propose AffineGlue to jointly perform feature matching and robust
estimation by leveraging a pool of one-to-many correspondences. Thus, it is
significantly less sensitive to matching ambiguities than using traditional
top-1 matches. AffineGlue significantly improves performance when applied on
top of popular feature detectors and matchers, such as SIFT or
SuperPoint+SuperGlue. Although the used solvers assume that the gravity
direction is known in both images, AffineGlue is so robust that the
$[0,-1,0]^{\text{T}}$ gravity prior works even on ScanNet, where it is only a
rough approximation with an avg. error of $24.8^{\circ}$ compared to the
actual vertical direction.

Limitations and Future Directions. One limitation is that most detectors and
matchers do not consider feature scale, orientation, and affine shape. The
only exception is the DoG + AffNet combination, where AffNet was trained on
DoG detections. We believe that creating end-to-end affine-covariant features
could boost the performance of an AffineGlue-based approach. Additionally,
considering the AC in the matching procedure could further improve accuracy,
_e.g_., by training SuperGlue on affine-aware descriptors.

## References

  * [1] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Seitz, and Richard Szeliski.  Building rome in a day.  Communications of the ACM, 54(10):105–112, 2011.
  * [2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In Computer Vision and Pattern Recognition, pages 5297–5307, 2016\.
  * [3] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk.  Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors.  In Computer Vision and Pattern Recognition, pages 5173–5182, 2017\.
  * [4] Daniel Barath, Luca Cavalli, and Marc Pollefeys.  Learning to find good models in RANSAC.  In Computer Vision and Pattern Recognition, pages 15744–15753, 2022\.
  * [5] Daniel Barath and Levente Hajder.  A theory of point-wise homography estimation.  Pattern Recognition Letters, 94:7–14, 2017.
  * [6] Daniel Barath and Levente Hajder.  Efficient recovery of essential matrix from two affine correspondences.  Transactions on Image Processing, 27(11):5328–5337, 2018.
  * [7] Daniel Barath and Levente Hajder.  Efficient recovery of essential matrix from two affine correspondences.  IEEE Trans. Image Process., 27(11):5328–5337, 2018.
  * [8] Daniel Barath, Levente Hajder, Dmytro Mishkin, and James Pritts.  Cvpr tutorial: Making affine correspondences work in camera geometry computation.  In Conference on Computer Vision and Pattern Recognition, 2022.
  * [9] Daniel Barath and Jiri Matas.  Graph-cut ransac: local optimization on spatially coherent structures.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4961–4974, 2021.
  * [10] Daniel Barath, Jiri Matas, and Jana Noskova.  Magsac: marginalizing sample consensus.  In Computer Vision and Pattern Recognition, pages 10197–10205, 2019\.
  * [11] Daniel Barath, Dmytro Mishkin, Ivan Eichhardt, Ilia Shipachev, and Jiri Matas.  Efficient initial pose-graph generation for global sfm.  In Computer Vision and Pattern Recognition, pages 14546–14555, 2021\.
  * [12] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas.  Magsac++, a fast, reliable and accurate robust estimator.  In Computer Vision and Pattern Recognition, pages 1304–1312, 2020\.
  * [13] Daniel Barath, Michal Polic, Wolfgang Förstner, Torsten Sattler, Tomas Pajdla, and Zuzana Kukelova.  Making affine correspondences work in camera geometry computation.  In European Conference on Computer Vision, pages 723–740. Springer, 2020.
  * [14] Daniel Barath and Gabor Valasek.  Space-partitioning RANSAC.  European Conference on Computer Vision, 2022.
  * [15] Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk.  Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters.  In International Conference on Computer Vision, 2019.
  * [16] Adam Baumberg.  Reliable feature matching across widely separated views.  In Computer Vision and Pattern Recognition, pages 1774–1781. IEEE Computer Society, 2000.
  * [17] P. R. Beaudet.  Rotationally invariant image operators.  In Proceedings of the 4th International Joint Conference on Pattern Recognition, 1978.
  * [18] Eric Brachmann and Carsten Rother.  Neural-guided ransac: Learning where to sample model hypotheses.  In International Conference on Computer Vision, pages 4322–4331, 2019.
  * [19] Capturing Realityn.  Realitycapture.
  * [20] Luca Cavalli, Viktor Larsson, Martin Ralf Oswald, Torsten Sattler, and Marc Pollefeys.  Handcrafted outlier detection revisited.  In European Conference on Computer Vision, 2020.
  * [21] Luca Cavalli, Marc Pollefeys, and Daniel Barath.  Nefsac: Neurally filtered minimal samples.  European Conference on Computer Vision, 2022.
  * [22] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan.  Aspanformer: Detector-free image matching with adaptive span transformer.  In European Conference on Computer Vision, pages 20–36. Springer, 2022.
  * [23] Rui Chen, Songfang Han, Jing Xu, and Hao Su.  Point-based multi-view stereo network.  In International Conference on Computer Vision, October 2019.
  * [24] Ondrej Chum and Jiri Matas.  Matching with prosac-progressive sample consensus.  In Computer Vision and Pattern Recognition, volume 1, pages 220–226. IEEE, 2005.
  * [25] Ondřej Chum and Jiří Matas.  Optimal randomized RANSAC.  Transactions on Pattern Analysis and Machine Intelligence, 30(8):1472–1482, 2008.
  * [26] Ondřej Chum, Jiří Matas, and Josef Kittler.  Locally optimized ransac.  In Joint Pattern Recognition Symposium, pages 236–243. Springer, 2003.
  * [27] Ondrej Chum, Tomas Werner, and Jiri Matas.  Two-view geometry estimation unaffected by a dominant plane.  In Conference on Computer Vision and Pattern Recognition. IEEE, 2005\.
  * [28] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In Computer Vision and Pattern Recognition, pages 5828–5839, 2017\.
  * [29] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Toward geometric deep slam.  arXiv preprint arXiv:1707.07410, 2017.
  * [30] D. Detone, T. Malisiewicz, and A. Rabinovich.  Superpoint: Self-Supervised Interest Point Detection and Description.  CVPRW Deep Learning for Visual SLAM, 2018.
  * [31] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In Computer Vision and Pattern Recognition workshops, pages 224–236, 2018.
  * [32] Yaqing Ding, Jian Yang, and Hui Kong.  An efficient solution to the relative pose estimation with a common direction.  In International Conference on Robotics and Automation, pages 11053–11059. IEEE, 2020.
  * [33] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  In Computer Vision and Pattern Recognition, 2019.
  * [34] Ivan Eichhardt and Daniel Barath.  Relative pose from deep learned depth and a single affine correspondence.  In European Conference on Computer Vision, pages 627–644. Springer, 2020.
  * [35] Jakob Engel, Thomas Schöps, and Daniel Cremers.  Lsd-slam: Large-scale direct monocular slam.  In European conference on computer vision, pages 834–849. Springer, 2014.
  * [36] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [37] J-M Frahm and Marc Pollefeys.  Ransac for (quasi-) degenerate data (qdegsac).  In Computer Vision and Pattern Recognition, volume 1, pages 453–460. IEEE, 2006.
  * [38] Yasutaka Furukawa, Brian Curless, Steven M Seitz, and Richard Szeliski.  Towards internet-scale multi-view stereo.  In Conference on Computer Vision and Pattern Recognition, pages 1434–1441. IEEE, 2010.
  * [39] Yasutaka Furukawa, Carlos Hernández, et al.  Multi-view stereo: A tutorial.  Foundations and Trends® in Computer Graphics and Vision, 9(1-2):1–148, 2015.
  * [40] Banglei Guan, Ang Su, Zhang Li, and Friedrich Fraundorfer.  Rotational alignment of imu-camera systems with 1-point ransac.  In Chinese Conference on Pattern Recognition and Computer Vision, pages 172–183. Springer, 2019.
  * [41] Banglei Guan, Ji Zhao, Zhang Li, Fang Sun, and Friedrich Fraundorfer.  Relative pose estimation with a single affine correspondence.  Transactions on Cybernetics, 2021.
  * [42] Levente Hajder and Daniel Barath.  Relative planar motion for vehicle-mounted cameras from a single affine correspondence.  In International Conference on Robotics and Automation, pages 8651–8657. IEEE, 2020.
  * [43] C. Harris and M. Stephens.  A combined corner and edge detector.  In Proceedings of the 4th Alvey Vision Conference, pages 147–151, 1988.
  * [44] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [45] Richard I Hartley.  In defense of the eight-point algorithm.  IEEE Transactions on pattern analysis and machine intelligence, 19(6):580–593, 1997.
  * [46] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448–456, Lille, France, 07–09 Jul 2015. PMLR.
  * [47] Heinly Jared, Johannes L Schonberger, Enrique Dunn, and Jan-Michael Frahm.  Reconstructing the world in six days.  In Computer Vision and Pattern Recognition, 2015.
  * [48] Yuhe Jin, Dmytro Mishkin, Anastasiia Mishchuk, Jiri Matas, Pascal Fua, Kwang Moo Yi, and Eduard Trulls.  Image matching across wide baselines: From paper to practice.  International Journal of Computer Vision, 2020.
  * [49] Mahzad Kalantari, Amir Hashemi, Franck Jung, and Jean-Pierre Guédon.  A new solution to the relative orientation problem using only 3 points and the vertical direction.  J. Math. Imaging Vis., 39(3):259–268, 2011.
  * [50] Abhishek Kar, Christian Häne, and Jitendra Malik.  Learning a multi-view stereo machine.  Advances in neural information processing systems, 30, 2017.
  * [51] Philip A Knight.  The sinkhorn–knopp algorithm: convergence and applications.  SIAM Journal on Matrix Analysis and Applications, 30(1):261–275, 2008.
  * [52] Zuzana Kukelova, Martin Bujnak, and Tomas Pajdla.  Automatic generator of minimal problem solvers.  In European Conference on Computer Vision, pages 302–315. Springer, 2008.
  * [53] Zuzana Kukelova, Joe Kileel, Bernd Sturmfels, and Tomas Pajdla.  A clever elimination strategy for efficient minimal solvers.  In Computer Vision and Pattern Recognition, pages 4912–4921, 2017\.
  * [54] Karel Lebeda, Jirı Matas, and Ondrej Chum.  Fixing the locally optimized ransac–full experimental evaluation.  In British machine vision conference, volume 2. Citeseer, 2012.
  * [55] Jongmin Lee, Yoonwoo Jeong, and Minsu Cho.  Self-supervised learning of image scale and orientation.  In 31st British Machine Vision Conference 2021, BMVC 2021, Virtual Event, UK. BMVA Press, 2021.
  * [56] David G. Lowe.  Distinctive image features from scale-invariant keypoints.  International Journal of Computer Vision (IJCV), 60(2):91–110, 2004.
  * [57] Simon Lynen, Bernhard Zeisl, Dror Aiger, Michael Bosse, Joel Hesch, Marc Pollefeys, Roland Siegwart, and Torsten Sattler.  Large-scale, real-time visual–inertial localization revisited.  The International Journal of Robotics Research, 39(9):1061–1084, 2020.
  * [58] Jiayi Ma, Junjun Jiang, Huabing Zhou, Ji Zhao, and Xiaojie Guo.  Guided locality preserving feature matching for remote sensing image registration.  IEEE transactions on geoscience and remote sensing, 56(8):4435–4447, 2018.
  * [59] J. Matas, O. Chum, M. Urban, and T. Pajdla.  Robust wide baseline stereo from maximally stable extrema regions.  In BMVC, pages 384–393, 2002.
  * [60] Krystian Mikolajczyk and Cordelia Schmid.  Scale & affine invariant interest point detectors.  International Journal of Computer Vision (IJCV), 60(1):63–86, 2004.
  * [61] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas.  Working Hard to Know Your Neighbor’s Margins: Local Descriptor Learning Loss.  In NeurIPS, 2017.
  * [62] D. Mishkin, F. Radenovic, and J. Matas.  Repeatability is Not Enough: Learning Affine Regions via Discriminability.  In European Conference on Computer Vision, 2018.
  * [63] Lionel Moisan, Pierre Moulon, and Pascal Monasse.  Automatic homographic registration of a pair of images, with a contrario elimination of outliers.  Image Processing On Line, 2:56–73, 2012.
  * [64] J. Molnár and D. Chetverikov.  Quadratic transformation for planar mapping of implicit surfaces.  Journal of Mathematical Imaging and Vision, 2014.
  * [65] Jorge J Moré.  The levenberg-marquardt algorithm: implementation and theory.  In Numerical analysis, pages 105–116. Springer, 1978.
  * [66] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  IEEE transactions on robotics, 31(5):1147–1163, 2015.
  * [67] D.R. Myatt, P.H.S. Torr, S.J. Nasuto, J.M. Bishop, and R. Craddock.  NAPSAC: High noise, high dimensional robust estimation-it’s in the bag.  In Proceedings of the British Machine Vision Conference, pages 44.1–44.10. BMVA Press, 2002.
  * [68] D. Nister.  An efficient solution to the five-point relative pose problem.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(6):756–770, 2004.
  * [69] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  In Proceedings of the IEEE international conference on computer vision, pages 3456–3465, 2017.
  * [70] Vojtech Panek, Zuzana Kukelova, and Torsten Sattler.  Meshloc: Mesh-based visual localization.  2022\.
  * [71] Michal Perdoch, Ondrej Chum, and Jiri Matas.  Efficient representation of local geometry for large scale object retrieval.  In Computer Vision and Pattern Recognition, pages 9–16, 2009.
  * [72] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Cnn image retrieval learns from bow: Unsupervised fine-tuning with hard examples.  In European conference on computer vision, pages 3–20. Springer, 2016.
  * [73] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  International Conference on Computer Vision, 2021.
  * [74] Rene Ranftl and Vladlen Koltun.  Deep fundamental matrix estimation.  In European Conference on Computer Vision, 2018.
  * [75] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  Transactions on Pattern Analysis and Machine Intelligence, 44(3), 2022.
  * [76] Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  R2d2: Repeatable and reliable detector and descriptor.  In NeurIPS, 2019.
  * [77] Clément Riu, Vincent Nozick, Pascal Monasse, and Joachim Dehais.  Classification performance of ransac algorithms with automatic threshold estimation.  In VISIGRAPP, volume 5, pages 723–733. Scitepress, 2022.
  * [78] Edward Rosten and Tom Drummond.  Machine learning for high-speed corner detection.  In European Conference on Computer Vision, ECCV’06, pages 430–443, Berlin, Heidelberg, 2006. Springer-Verlag.
  * [79] E. Rublee, V. Rabaud, K. Konolidge, and G. Bradski.  ORB: An Efficient Alternative to SIFT or SURF.  In International Conference on Computer Vision, 2011.
  * [80] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In Computer Vision and Pattern Recognition, pages 4938–4947, 2020\.
  * [81] Torsten Sattler, Bastian Leibe, and Leif Kobbelt.  Improving image-based localization by active correspondence search.  In European conference on computer vision, pages 752–765. Springer, 2012.
  * [82] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In Computer Vision and Pattern Recognition, pages 8601–8610, 2018\.
  * [83] Davide Scaramuzza.  1-point-RANSAC structure from motion for vehicle-mounted cameras by exploiting non-holonomic constraints.  International journal of computer vision, 95(1):74–85, 2011.
  * [84] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Computer Vision and Pattern Recognition, pages 4104–4113, 2016\.
  * [85] Rajvi Shah, Vanshika Srivastava, and PJ Narayanan.  Geometry-aware feature matching for structure from motion applications.  In Winter Conference on Applications of Computer Vision, pages 278–285. IEEE, 2015.
  * [86] Charles V. Stewart.  Minpran: A new robust estimator for computer vision.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(10):925–938, 1995.
  * [87] Henrik Stewenius, Christopher Engels, and David Nistér.  Recent developments on direct relative orientation.  ISPRS Journal of Photogrammetry and Remote Sensing, 60(4):284–294, 2006.
  * [88] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  In Computer Vision and Pattern Recognition, pages 8922–8931, 2021\.
  * [89] Weiwei Sun, Wei Jiang, Andrea Tagliasacchi, Eduard Trulls, and Kwang Moo Yi.  Attentive context normalization for robust permutation-equivariant learning.  In Conference on Computer Vision and Pattern Recognition, 2020.
  * [90] Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas.  Sosnet: Second order similarity regularization for local descriptor learning.  In Computer Vision and Pattern Recognition, 2019.
  * [91] Giorgos Tolias, Yannis Avrithis, and Hervé Jégou.  Image search with selective match kernels: aggregation across single and multiple images.  International Journal of Computer Vision, 116(3):247–261, 2016\.
  * [92] P. H. S. Torr and A. Zisserman.  MLESAC: A new robust estimator with application to estimating image geometry.  Computer Vision and Image Understanding (CVIU), 2000.
  * [93] Michał J. Tyszkiewicz, Pascal Fua, and Eduard Trulls.  Disk: Learning local features with policy gradient.  In NeurIPS, 2020.
  * [94] C. Varytimidis, K. Rapantzikos, and Y. Avrithis.  Wash: Weighted $\alpha$-shapes for local feature detection.  In European Conference on Computer Vision 2012, 2012.
  * [95] Abraham Wald.  Sequential Analysis.  John Wiley and Sons, 1st edition, 1947.
  * [96] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen.  Matchformer: Interleaving attention in transformers for feature matching.  arXiv preprint arXiv:2203.09645, 2022.
  * [97] Kwang Moo Yi*, Eduard Trulls*, Yuki Ono, Vincent Lepetit, Mathieu Salzmann, and Pascal Fua.  Learning to find good correspondences.  In Conference on Computer Vision and Pattern Recognition, 2018.
  * [98] K. M. Yi, Y. Verdie, P. Fua, and V. Lepetit.  Learning to Assign Orientations to Feature Points.  In Computer Vision and Pattern Recognition, 2016.
  * [99] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao.  Learning two-view correspondences and geometry using order-aware network.  International Conference on Computer Vision, 2019.
  * [100] Chen Zhao, Yixiao Ge, Feng Zhu, Rui Zhao, Hongsheng Li, and Mathieu Salzmann.  Progressive correspondence pruning by consensus learning.  In International Conference on Computer Vision, 2021.
  * [101] Siyu Zhu, Runze Zhang, Lei Zhou, Tianwei Shen, Tian Fang, Ping Tan, and Long Quan.  Very large-scale global sfm by distributed motion averaging.  In Computer Vision and Pattern Recognition, pages 4568–4577, 2018\.