# LFM-3D: Learnable Feature Matching Across Wide Baselines Using 3D Signals

Arjun Karpur† Guilherme Perrotta∗ Ricardo Martin-Brualla† Howard Zhou† André
Araujo†

†Google Research
<EMAIL_ADDRESS>∗University of
Campinas (UNICAMP)
<EMAIL_ADDRESS>
###### Abstract

Finding localized correspondences across different images of the same object
is crucial to understand its geometry. In recent years, this problem has seen
remarkable progress with the advent of deep learning based local image
features and learnable matchers. Still, learnable matchers often underperform
when there exists only small regions of co-visibility between image pairs
(_i.e_. wide camera baselines). To address this problem, we leverage recent
progress in coarse single-view geometry estimation methods. We propose LFM-3D,
a Learnable Feature Matching framework that uses models based on graph neural
networks, and enhances their capabilities by integrating noisy, estimated 3D
signals to boost correspondence estimation. When integrating 3D signals into
the matcher model, we show that a suitable positional encoding is critical to
effectively make use of the low-dimensional 3D information. We experiment with
two different 3D signals - normalized object coordinates and monocular depth
estimates - and evaluate our method on large-scale (synthetic and real)
datasets containing object-centric image pairs across wide baselines. We
observe strong feature matching improvements compared to 2D-only methods, with
up to +6% total recall and +28% precision at fixed recall. We additionally
demonstrate that the resulting improved correspondences lead to much higher
relative posing accuracy for in-the-wild image pairs, with a more than 8%
boost compared to the 2D-only approach.

##  1 Introduction

Figure 1:  We propose LFM-3D, a novel learnable method for local feature
matching leveraging 3D information. Infusing local feature matching with 3D
signals enables accurate estimation of correspondences across very wide
baselines, where conventional methods (SIFT + ratio test [21]) and even recent
ones (SuperPoint + SuperGlue [34]) fail – we represent correct matches with
green lines, and incorrect ones with red lines. Here, our method incorporates
3D normalized object coordinates as part of a graph neural network matcher,
which significantly boosts the feature association process.

Matching images of the same object or scene is a core task in computer vision,
since it enables the recovery of 3D structure and camera poses. Many
applications depend on these capabilities, ranging from augmented/virtual
reality to novel view synthesis from in-the-wild image collections. Central to
image matching are local image features and associated algorithms. ††∗This
work was completed while Guilherme was an intern at Google. Traditionally,
hand-crafted techniques such as SIFT [21] or SURF [3] were used to detect
keypoints and describe them via local gradient statistics. Putative feature
matches were then obtained via simple nearest neighbor search, or its variants
(_e.g_., Lowe’s ratio test [21]), and fed into robust estimation methods such
as RANSAC [15].

In the past few years, deep learning based techniques have been proposed to
replace hand-crafted methods in this area. Several papers initially proposed
to learn keypoint detectors [51, 2] or local descriptors [26, 39, 40]. Later,
techniques were introduced to jointly model these two stages [9, 32, 44],
improving performance with end-to-end learning. More recently, learnable
feature matcher models were proposed to replace hand-crafted matching
heuristics and obtain improved putative correspondences [50, 34, 37].

Unfortunately, these methods still encounter difficulties in many cases.
Although learnable matchers integrate keypoint locations and global context
into correspondence estimation, wide baselines, where small co-visibility
regions appear, often hinder performance, as illustrated in Fig. 1. In this
paper, we propose to address such challenging situations by integrating
estimated 3D signals to help guide local feature matching even further. We
propose using prior learned knowledge to infer rough estimates of the 3D
structure of the underlying object. Then, along with the 2D keypoint
locations, these 3D signals can help disambiguate matches when pixel-based
descriptors are uninformative.

Depending on the availability of images and 3D models for given objects of
interest, different types of 3D signals can be used. In the case where a
large-scale 3D model repository is available, we learn a class-specific
Normalized Object Coordinate Space [47] to predict 3D coordinates for points
on the object in a canonical orientation. On the other hand, in the long-
tailed case where no such data is available, we resort to class-agnostic
monocular depth estimates [31, 30] which can still provide helpful 3D hints
that improve matching. We perform experiments on large-scale datasets for the
_shoe_ and _camera_ object classes, both in controlled and in-the-wild capture
scenarios. In more detail, our contributions are as follows.

Contributions. (1) A novel method for local feature matching that can
integrate different types of 3D information to help guide correspondence
estimation, within a learnable framework that is a natural and simple
extension to SuperGlue [34]. 3D signals are associated with each local feature
and input alongside descriptors and 2D positions into a graph neural network
model that predicts correspondences. This method improves feature matching
recall by more than 6%, and precision by up to 28%. Example results are
highlighted in Fig. 1. (2) We demonstrate that suitable positional encodings
are required for this task in order to have 3D information actually help;
simple encodings used in the learnable feature matching literature do not help
much. (3) As an example application of the improved correspondences, we
present significant improvements in the downstream task of object relative
pose estimation based on the improved matches. In the large-scale Objectron
dataset [1] containing in-the-wild images, relative pose estimation for shoes
improves significantly, by up to $8.6$% compared to the 2D-only approach. We
also outperform a direct dense PnP method based on 3D coordinates, by up to
$10.4$%.

##  2 Related Work

Traditional local feature extraction and matching. Traditionally, hand-crafted
methods, such as SIFT [21], SURF [3] or others [42, 5, 4], have been used to
find correspondences between images. Such keypoint detection approaches
generally rely on finding salient low-level patterns such as blobs or corners,
with representative methods being Difference-of-Gaussians [21], Harris-Laplace
or Hessian-Affine [24]. Local description in this case is mainly based on
gradient orientation statistics for small image patches around the keypoint.
In this conventional pipeline, associating local features across images
generally builds on top of nearest neighbor matching, using Lowe’s ratio test
[21] or mutual consistency checks, and possibly additionally leveraging
consensus methods [43, 35].

Deep learning based local features. As the interest in deep learning methods
for computer vision grew, hand-crafted feature extraction gave way to learned
techniques. As a first step, researchers proposed to independently learn
either the keypoint detection [46, 51, 27, 2] or the keypoint description [26,
22, 39, 40, 36, 48] stages. This led to systems which were partially learned,
but still relied on hand-crafted components. A more recent trend is the
unified modeling of detection and description, which enables end-to-end
learning [49, 28, 9, 29, 32, 12, 44], leading to stronger results in image
matching related tasks. These methods may be trained using different levels of
supervision: supervised at the feature level where the correspondence labels
are computed from Structure-from-Motion pipelines [49, 29, 32, 12, 44], weaker
supervision at the level of relative camera poses [48] or image-level labels
[28], or finally fully self-supervised with synthetic image transformations
[9]. Learned local features improve upon hand-crafted methods in several
respects, _e.g_.: the detection process extracts much more discriminative
keypoints instead of hand-crafted low-level corners or blobs; the descriptor
achieves more robust illumination and viewpoint invariance due to deep
semantic representations. Still, wide baseline matching and the presence of
repeated patterns (_e.g_., building windows) may confuse these techniques and
lead to poor correspondences.

Learnable feature matching. One of the latest trends in the area of local
image features is to additionally enable the learning of the feature matching
process. This goes beyond learned outlier filtering approaches [50, 52].
SuperGlue [34] proposed graph neural network models in conjunction with
optimal transport optimization in order to effectively associate sparse local
features and reject outliers. This method takes in sets of local features from
two images, comprising local descriptors and 2D positions, and learns feature
matching patterns with a data-driven approach. A follow-up method [6] proposed
to improve matching efficiency with reliable seed matches to initialize the
process. The learnable matching framework has also evolved to perform matching
in a dense manner [7, 37], which avoids issues with potentially noisy keypoint
detections at the cost of higher memory requirements. In this work, we focus
on the sparse case and extend the learnable feature matching model to consider
3D information at the keypoint level, allowing the framework to reason beyond
2D signals.

Feature matching leveraging 3D information. Most local feature methods do not
consider 3D information in the matching process. Closest in spirit to our work
is the method of Toft _et al_. [41], where monocular depth estimation is used
to rectify perspective distortions, improving the quality of extracted local
features. In contrast, our work can leverage different types of 3D information
in a learnable framework, which enables more precise matching. Another recent
work in this area by Ma _et al_. [23] relies on human poses as 3D priors in
order to disambiguate occlusions during matching. Our method shares some
commonalities with it in the sense that both rely on some prior about the 3D
shape of an object depicted in a 2D image. However, we differ from them as our
goal is to improve local feature representations for matching of visible
parts, as opposed to their focus on associating invisible points for bundle
adjustment.

##  3 LFM-3D

###  3.1 Background

We start by introducing the necessary background on learnable sparse feature
matching, normalized object coordinates, and monocular depth estimation.

Learnable sparse feature matching. Traditional matching methods rely solely on
image region descriptors to propose correspondences. Learnable matchers
improve upon these techniques by using additional information during the
matching process, such as keypoint locations and global image context. Our
method builds on top of SuperGlue [34], a landmark learnable sparse matching
network, so we briefly introduce it here along with the required notation.

SuperGlue [34] is a graph neural network (GNN) model architecture that
predicts correspondences between a pair of sparse keypoint sets. Denote local
features extracted from an image as $({\bf p},{\bf d})$, where
$\mathbf{p}_{i}:=(x,y,c)_{i}$ corresponds to the 2D position $x,y$ and
confidence score $c$ for the $i$-th local feature, and
$\mathbf{d}_{i}\in\mathbb{R}^{D}$ is the $i$-th local descriptor. A keypoint
encoder combines $\mathbf{p}_{i}$ and $\mathbf{d}_{i}$ to produce a new
representation $\mathbf{x}_{i}$ which will be used for matching as follows:

$\mathbf{x}_{i}=\mathbf{d}_{i}+\text{MLP}\left(\mathbf{p}_{i}\right)$ (1)

where MLP denotes a multi-layer perceptron that performs learnable positional
encoding. Alternating layers of self-attention (within a single keypoint set)
and cross-attention (between keypoint sets of two images) iteratively refine
$\mathbf{x}_{i}$ by providing global context for both images to the local
region.

Training uses ground truth correspondence supervision at the local feature
level, and an optimal transport optimization problem is leveraged to solve for
a partial assignment between the two feature sets. Ground truth
correspondences are recovered for pairs of images using either known
homographies or complete camera and depth information.

Normalized object coordinates. The Normalized Object Coordinate Space (NOCS)
[47] is a 3D space defined over the unit cube that can be used to represent
the geometry of all instances of an object class in a canonical orientation.
Given an image of an unseen object instance, a NOCS map can be computed that
predicts 3D normalized coordinates $\mathbf{n}_{i}\in\mathbb{R}^{3}$ per pixel
with respect to the canonical reference frame. Note that a prerequisite to
exploit NOCS is to register all objects in the category in the same coordinate
space. From the NOCS estimates, one can solve for the 6D object pose of the
instance with respect to the canonical orientation by using the 2D-3D
correspondences between image pixels and NOCS points to solve the Perspective-
n-Point (PnP) problem [19, 18]. However, we observe in our experiments that
fine-grained pose estimation accuracy is highly sensitive to the quality of
the predicted NOCS map.

Monocular depth estimation. Monocular Depth Estimation (MDE) [14, 16] is the
task of predicting the depth value of each pixel given a single RGB image as
the input. In most modern applications, the output is a dense inverse depth
map of the same resolution as the input image. Each pixel takes on a value
proportional to the disparity $1/d$, where $d$ is the estimated depth of the
pixel. Among various other reasons, this formulation better handles depth
values at large distances. Compared to depth obtained from metric-based depth
sensors (stereo cameras, LIDAR, etc.), MDE suffers from scale ambiguity, which
makes it hard to be used directly for pose estimation. However, for our
purpose, we found that MDE provides sufficient 3D signals to boost
correspondence estimation, and we use the publicly available DPT [30] model
for MDE.

###  3.2 Learning to match using 3D signals

In the following, we present our method to integrate 3D signals in a learnable
manner for improved feature matching, which is illustrated in Fig. 2.

Figure 2:  Block diagram of an instance of the proposed LFM-3D system, which
uses NOCS [47] maps for 3D signals. We extract local features and normalized
object coordinates (NOCS) from each image. The NOCS maps are visualized by
mapping XYZ to RGB. The NOCS 3D coordinates undergo positional encoding and
are combined with local features in order to generate 3D-infused local feature
embeddings. A graph neural network is then applied on these to propose
correspondences. Our method can find correspondences across images under very
wide baselines, thanks to the 3D information leveraged from NOCS. Besides
NOCS, in this work we also instantiate the LFM-3D model with monocular depth
estimates (MDE), which would follow the same process by changing NOCS maps to
MDE maps.

Notation for 3D signals. Since many relevant 3D signals can be obtained as a
dense feature map, to simplify the presentation, we assume that estimated
information about the structure of the underlying object is extracted in this
form. Note, though, that sparse 3D signals could also naturally be
incorporated into our framework. Given an image
$\textit{I}\in\mathbb{R}^{H\times W}$, the corresponding feature map
$N\in\mathbb{R}^{H\times W\times V},V\geq 1$ relates each image pixel to some
vector $\mathbf{n}_{i}$ that encodes local information about the object’s
geometry. Given sufficient training data on the object class (i.e. 3D models),
we can learn a strong prior over the object’s geometry. For example, a known
object class with an accurate NOCS regression model available allows us to
associate each pixel with an (estimated) 3D point on the object in the
canonical orientation ($V=3$). Under weaker assumptions, monocular depth
estimates may be used to relate each pixel to an estimated distance from the
capturing camera ($V=1$).

Enhancing local features with 3D information. Given the feature map $N$, we
use bilinear interpolation to look up the prediction for a local feature
coordinate. Thus, each local feature $i$ has information about its 2D pixel
coordinates $\mathbf{p}_{i}$, local image content $\mathbf{d}_{i}$, and
predicted 3D signal $\mathbf{n}_{i}\in\mathbb{R}^{V}$.

Extending SuperGlue to handle 3D signals. The 3D signals for each local
feature can be directly used to improve each keypoint’s matching
representation:

$\mathbf{x}^{\prime}_{i}=\mathbf{d}_{i}+\text{MLP}_{\text{2D}}\left(\mathbf{p}_{i}\right)+\text{MLP}_{\text{3D}}\left(\mathbf{n}_{i}\right)$
(2)

where $\text{MLP}_{\text{2D}}$ and $\text{MLP}_{\text{3D}}$ denote separate
multi-layer perceptrons that act on the 2D keypoint information and the 3D
signals, respectively. This 3D-infused representation
$\mathbf{x}^{\prime}_{i}$ can be directly used in the GNN with the same
architecture and loss functions to improve feature matching. Note that summing
up the features instead of concatenating allows for pre-training on datasets
where no 3D information is available, as described below.

Improved positional encoding. However, MLPs are know to have a spectral bias
to learn overly smooth functions [38]. This is particularly prevalent in
models that learn function over a low dimensional space, such as from NOCS or
inverse depth. Positional encoding [38, 45] lifts the input representation to
a higher dimension space using periodic functions of increasing frequency. In
our model, we use them to encode $\mathbf{n_{i}}$ before passing them onto the
MLP. Our final representation for each local feature is then:

$\hat{\mathbf{x}}_{i}=\mathbf{d}_{i}+\text{MLP}_{\text{2D}}\left(\mathbf{p}_{i}\right)+\text{MLP}_{\text{3D}}(\text{PE}\left(\mathbf{n}_{i}\right))$
(3)

where PE stands for positional encoding. We use 10 frequencies with an
implementation following the one of NeRF [25]. We refer to the new keypoint
representations $\hat{\mathbf{x}}$ as 3D-infused local feature embeddings, or
3D embeddings for short.

Multi-stage training. Limitations in object-centric data with the necessary
match-level annotations may cause the model to generalize poorly to different
domains (e.g. from synthetic to real-world images). However, we note that
there exists many multi-view and Structure-from-Motion datasets for outdoor
scenes that can be used to initialize some layers of LFM-3D. We bootstrap the
LFM-3D model by training the $\text{MLP}_{\text{2D}}$ and GNN layers of the
matcher model from scratch for many iterations on synthetic homography and
outdoor scene multi-view datasets (similar to [34]). Once the model has a
strong understanding of the 2D correspondence estimation task, we introduce
the $\text{MLP}_{\text{3D}}$ encoding layer and finetune all matcher-model
layers jointly on object-centric data augmented with 3D estimates.

##  4 Experiments

###  4.1 Datasets

We focus our experiments on object-centric image pairs taken from varying
angles. We use synthetic renderings of 3D models for training, and evaluate on
both synthetic and real-world image pairs. Table 1 contains our dataset
statistics.

Object class | Dataset variant | Angles (∘) | # Scenes / Pairs
---|---|---|---
_Shoe_ | GSO-Train |  $15$-$75$ |  $327$ / $490$k
GSO-Eval |  $90$-$120$ |  $30$ / $4479$
Objectron |  $90$-$120$ |  $423$ / $6840$
_Camera_ | GSO-Train |  $15$-$75$ |  $40$ / $60$k
GSO-Eval |  $90$-$120$ |  $7$ / $1050$
Objectron |  $90$-$120$ |  $163$ / $5087$

Table 1: Training and evaluation dataset statistics. We leverage the Google
Scanned Objects (GSO) [11] and Objectron [1] datasets.

Google Scanned Objects [11]. We use the high-quality scanned models with
realistic textures provided by the Google Scanned Objects (GSO) dataset to
train the learnable matcher components of LFM-3D. Within GSO, we focus on the
_shoe_ and _camera_ object categories, of which there are about 360 and 47
entries, respectively, after filtering bad quality and duplicated models. Note
that the 3D models were also prealigned during the capture process. To define
the canonical coordinate space, we simply scale all the models so they have
the same length (about $30cm$). We render each object in 500 random poses,
with a distribution that tries to match the one of real photos (instances are
likely upright, etc). Additionally, we randomly modify the focal length and
distance to the object and generate partial views by applying random crops.
Each rendered frame also provides ground truth camera matrices and depth maps.
To generate ground truth correspondence data, we extract local features from
each image and use the camera/depth information to project a keypoint set from
one image onto the other. Following [34], we select mutual nearest neighbors
in reprojected pixel coordinate space and consider keypoints within 3 pixels
as ground truth matches.

Objectron [1]. The Objectron dataset contains object-centric videos with wide
viewpoint coverage for nine object categories, including _shoe_ and _camera_.
Each video frame is labeled with camera intrinsics and extrinsics, and shows
the instance in a real-world scene. To overcome the domain gap between
synthetic and real-world images, we filter out all frames with $>1$ object
instance and use an off-the-shelf segmentation model to limit extracted
features to the instance segmentation mask. The lack of metric depth data
prevents us from training on Objectron frame pairs, but we leverage the freely
available camera matrices for evaluation purposes.

Evaluation pairs. For both source datasets, we sample image pairs at wide
baselines to evaluate matcher performance in challenging scenarios; Table 1
details each variant and its statistics. Specifically, we select two frames
from the same scene and use the angular distance between the cameras’ optical
axes to filter for a specific camera baseline.

###  4.2 Acquiring estimated 3D signals

Given the flexible nature of LFM-3D, we provide experiments on two types of 3D
estimates: NOCS maps and MDE inverse depth maps. The following section details
how we retrieve these predicted 3D signals for each image pair.

Normalized object coordinates. We opt to train a new NOCS map regression
network for the _shoe_ object class, which contains enough 3D models for
learning class-specific object coordinates. As training data, we use the $327$
shoe models to produce renderings at resolutions of $256\times 256$. We
randomly sample cameras while oversampling images with the object’s up-vector
aligned to the image’s up-vector, and sample various crops, distances to the
object and focal lengths, all while maintaining the object extent constant.
The network is a reduced-capacity version of Imagen’s U-Net [33], with a total
of $28$M parameters, with fewer attention heads and features per layer. Our
model regresses the NOCS coordinates directly, in contrast to the
classification approach [17] used in [47]. Unlike [47], we output NOCS
coordinates at the same resolution as the input image. We train the model with
batch size $96$ for $300$k iterations on 8 NVIDIA Tesla V100 GPUs.

At inference time, given an image, we first apply an off-the-shelf instance
segmenter for the target object category. Using the mask, we produce a tight
crop around the object and pass it to the feed-forward network that estimates
NOCS coordinates per pixel. On the synthetic dataset, the network is able to
achieve a regression error of $1.4cm$ on a test set of unseen shoe models. We
show qualitative results of running NOCS on Objectron in Figure 3. Note that
NOCS is only trained on synthetic renders from Google Scanned Objects, but is
able to generalize to real images. We also note that the network fails for
less common shoes (fifth column), as the dataset contains mostly sneaker type
shoes.

Input |  |  |  |  |
---|---|---|---|---|---
Alpha
NOCS
Figure 3:  Qualitative results of our trained NOCS model on Objectron. Top:
input image, middle: segmentation from off-the-shelf instance segmenter,
bottom: NOCS rendering with axis aligned grid overlaid. Note that the NOCS
model was trained only on synthetic renderings from Google Scanned Objects.

Monocular depth. We use an off-the-shelf, class-agnostic monocular depth
estimation model to predict depth maps for the _camera_ object class, to study
the long-tailed case where a large repository of 3D models is not available.
We use the publicly available DPT [30] model, which is based on a Vision
Transformer [10] backbone and trained on a large-scale meta-dataset across
many image domains [31]. Following convention, we directly use the output
inverse-depth maps to better handle depth values at large distances from the
camera [8]. Please see the appendix for qualitative depth estimate examples.

###  4.3 Experimental setup

Object classes. As previously mentioned, we focus our experiments on two
object classes that represent two separate use cases of LFM-3D: _shoes_ with
learned class-specific NOCS estimates, and _cameras_ with class-agnostic
monocular depth estimates.

Metrics. We report results on match-level statistics for the Google Scanned
Objects dataset. From a soft assignment matrix representing correspondence
scores between each keypoint pair, we can threshold these proposals to recover
a predicted correspondence set. We calculate precision/recall at the match
level as the ratio of correct matches (precision) and ratio of ground truth
matches recovered (recall) for a given correspondence prediction set. We
utilize fixed keypoint sets with known camera and depth to define recall.

Additionally, we evaluate correspondences on the downstream task of relative
pose estimation, which is critical to many 3D reconstruction tasks. We
estimate the essential matrix $E^{\prime}$ that relates the two cameras using
known camera intrinsics and predicted correspondences, and recover a relative
rotation prediction. We calculate the relative rotation error between the
predicted and ground truth rotation matrices using Rodrigues’ formula and
report accuracy within $5^{\circ},10^{\circ},$ and $15^{\circ}$ of error.

(a) Shoe ($90^{\circ}$\- $120^{\circ}$)

(b) Camera ($90^{\circ}$\- $120^{\circ}$)

Figure 4: Correspondence-level precision/recall curves for the Google Scanned
Objects evaluation datasets. We compare ablated versions of our LFM-3D method
to highlight each component.

Compared techniques. We compare LFM-3D to popular 2D-based sparse matching
techniques, such as SIFT [21] features with heuristics-based matchers and
SuperPoint [9] features with SuperGlue correspondence prediction. For _shoe_
relative pose estimation, we can also compare against sparse matching between
2D keypoints and their 3D NOCS estimates (_i.e_. Sparse PnP), and with dense
matching between all 2D pixel coordinates and their 3D NOCS estimates (_i.e_.
Dense PnP). We compare against additional techniques in the appendix.

| _Shoe_ | _Camera_
---|---|---
Method |  Acc@$5^{\circ}$ |  Acc@$10^{\circ}$ |  Acc@$15^{\circ}$ |  Acc@$5^{\circ}$ |  Acc@$10^{\circ}$ |  Acc@$15^{\circ}$
SIFT + MNN | 4.0 | 7.8 | 11.8 | 4.6 | 9.0 | 13.2
SIFT + NN/Ratio | 4.4 | 8.7 | 13.2 | 4.3 | 9.0 | 13.0
NOCS + Sparse-PnP | 4.8 | 18.4 | 33.4 | - | - | -
NOCS + Dense-PnP | 4.2 | 18.9 | 34.9 | - | - | -
SuperPoint + MNN | 3.4 | 7.1 | 11.1 | 2.7 | 5.9 | 8.8
SuperPoint + NN/Ratio | 4.0 | 7.8 | 11.9 | 3.0 | 6.6 | 10.2
SuperPoint + SG-MegaDepth | 2.6 | 5.6 | 8.6 | 2.7 | 5.5 | 8.6
SuperPoint + SG-Finetuned | 11.0 | 22.2 | 31.6 | 10.6 | 21.4 | 29.2
SuperPoint + LFM-3D (Ours) | 14.3 | 29.3 | 40.2 | 11.8 | 22.4 | 30.7
Table 2: Relative pose estimation results on Objectron sampled video frames.
We calculate the error between the ground truth rotation matrix and recovered
relative rotation matrix from essential matrix estimation. We report accuracy
within a given error threshold (in degrees), as defined by Rodrigues’ formula.
Note that we use estimated NOCS [47] maps as inputs to LFM-3D for the _shoe_
object category and estimated inverse depth maps from DPT [30] for _camera_.
We use all predictions (threshold $0.0$) when estimating the essential matrix.

Architecture and training details. Following [34], we use SuperPoint local
features [9] in our learnable matching framework. We initialize the 2D
keypoint encoder, graph neural network, and optimal transport layers from a
2D-only matching model pretrained on synthetic homographies and Megadepth [20]
image pairs, following the recipe described in [34]. The model extracting 3D
signals (_i.e_., NOCS or MDE) is frozen when training the LFM-3D matching
layers. We mirror the $\text{MLP}_{3D}$ architecture after $\text{MLP}_{2D}$,
but increase the number of parameters in each layer as follows:
$[64,64,128,128,D]$, where $D$ is the local feature descriptor
($\mathbf{d}_{i}$) dimension. An initial learning rate of
$8\text{\times}{10}^{-5}$ is used until 65k iterations, after which the
learning rate decays by a factor of $0.999995$ per iteration. The model was
trained for $750000$ iterations with a batch size of 56 on the same set of
GPUs as stated above. LFM-3D is implemented natively in Tensorflow 2 from
scratch, and we use a public Tensorflow reproduction of Superpoint.

(a) Google Scanned Objects wide-baseline pairs.

(b) Objectron wide-baseline image pairs.

|  |
---|---|---
(c) Failure cases for LFM-3D.

Figure 5: Qualitative results for our LFM-3D method. For all image pairs, we
show predicted correspondences with confidence threshold $0.2$. Note that an
absence of correspondence lines means that the model found no matches. (a)
Correct matches are shown in green and incorrect matches ($>3$ pixel error)
are shown in red. (b) & (c) Ground truth isn’t available, so we show matches
in randomized colors.

###  4.4 Results

Feature-level matching & model ablations. Figure 4 presents precision/recall
curves for LFM-3D on the Google Scanned Objects dataset for the _shoe_ and
_camera_ object classes. We compare LFM-3D against model variants that allow
us to ablate the importance of the different components in our system:
finetuning, usage of 3D signals, and improved positional encoding. For each
method, we extract a maximum of 1024 features and use ground truth camera and
depth information to label known correspondences. To generate the curves, we
sweep over confidence values in $[0,1]$ and filter match predictions from the
soft assignment output. We use the following terminology for variants:

  * •

SG-MegaDepth \- Default SuperPoint + SuperGlue method trained on synthetic
homography pairs and MegaDepth [20] outdoor scene pairs.

  * •

SG-Finetuned \- Default SuperPoint + SuperGlue model, initialized with SG-
MegaDepth and finetuned on Google Scanned Objects rendered pairs.

  * •

SG + NOCS/MDE \- Finetuned SuperPoint + SuperGlue model, with an additional
input encoding layer for 3D signals per keypoint (Eq. (2)). As described
above, we use points in NOCS space for _shoe_ and inverse depth estimates for
_camera_.

We observe that LFM-3D outperforms all baseline methods, with the most
significant improvements being in maximum recall (no confidence thresholding).
The NOCS-based LFM-3D outperforms SG-Finetuned on recall by $6.2\%$ and at
maximum recall for SG-Finetuned ($44.3\%$), it sees a $28.9\%$ increase in
precision. The significant gap between SG + MDE and LFM-3D (MDE) indicates the
importance of properly encoding low-dimensional 3D signals. An effective
positional encoding can enhance the model’s capability to integrate 3D
information. Lastly, we note a large performance gap between the baseline SG-
MegaDepth method and all methods finetuned on synthetic renderings.

Relative pose estimation. We evaluate relative pose estimation on the
Objectron [1] dataset, by sampling pairs of video frames with viewpoint
changes (_i.e_., camera axes angle distances) between
$90^{\circ}$-$120^{\circ}$. We limit keypoint extraction to a segmentation
mask (instance segmentation for shoes, and foreground segmentation for
cameras) to minimize the domain gap. Results are presented in Table 2; as
before, the _shoe_ LFM-3D model uses estimated NOCS maps and the _camera_
LFM-3D model uses estimated depth maps. We compare LFM-3D against conventional
and state-of-the-art sparse feature matching techniques. In the _shoe_ case,
where a NOCS model is available, we also compare to sparse and dense PnP
methods based on NOCS.

The LFM-3D model based on NOCS outperforms all other methods for the _shoe_
case. We observe a large gap between our method and the finetuned 2D-only
SuperPoint+SuperGlue: by $8.6$% for Acc$@15^{\circ}$. More interestingly, we
also outperform NOCS-based PnP variants by large margins (up to $10.4\%$
against the dense version), which shows the limitations of directly using
estimated NOCS maps for relative pose prediction.

Similarly, our depth-based LFM-3D model outperforms all methods in the
_camera_ case. Although we observe a less drastic improvement over the 2D-only
techniques, we note that our method makes no further assumptions about the
query image pair and uses predictions from an off-the-shelf MDE model to
provide up to $1.5\%$ improvement, compared to the finetuned 2D-only
SuperPoint+SuperGlue.

Qualitative results. Figure 5 shows how LFM-3D proposes improved
correspondences between images with relatively small overlapping visible
regions. SG-MegaDepth suffers from low confidence predictions under very-wide
baselines, but still correctly predicts many correspondences for minor
viewpoint changes (Figure 5(a), bottom left). Finetuning SuperGlue on domain-
specific data results in more high-confidence predictions, but adding 3D
signals with positional encoding leads to even better results, with more
correct matches and a wider distribution of proposed correspondences across
the image and the object’s geometry (Figure 5(b)), which simplifies essential
matrix estimation.

Limitations and failure scenarios. The disparity between LFM-3D performance on
the two object classes shows that the model can be sensitive to the choice of
3D signal. For long-tail objects that lack large-scale 3D data, one must rely
on a less informative 3D signal. Figure 5(c) shows examples where LFM-3D still
struggles to match reliably. In particular, we observe that feature-less
objects (first column) negatively affect local feature extractors and 3D
signal estimators alike. Irregular geometries (second column) can also confuse
the NOCS estimation model. And out-of-distribution geometries not shown in the
training set (camera strap - third column) can match poorly at inference time.

##  5 Conclusions

In this work, our main contribution is a novel method for local feature
matching that integrates estimated 3D signals to guide correspondence
estimation – leading to high-quality feature associations under wide baseline
conditions. Experiments indicate that the way the 3D signal is incorporated
into the model matters, with improvements observed when using positional
encodings based on periodic functions. Our enhanced feature correspondences
enable a significant boost in performance for relative pose estimation of in-
the-wild images. Based on the quantitative and qualitative evidence we
gathered, one can conclude that 3D information can greatly improve the
estimation of image correspondences.

## Appendix

|  |  |  |  |
---|---|---|---|---|---
Figure 6: Visualizations of monocular depth estimates on camera Objectron
images. Inverse depth values are generated using the general-purpose DPT [30]
monocular depth estimation model.

##  Appendix A Examples of monocular depth estimates

Figure 6 presents qualitative examples from the general-purpose, pre-trained
monocular depth estimation model, DPT [30]. We use the published model weights
available on their github.

##  Appendix B Comparison to additional baselines

| _Shoe_
---|---
Method |  Acc@$5^{\circ}$ |  Acc@$10^{\circ}$ |  Acc@$15^{\circ}$
Official SP+SG release | 2.3 | 4.8 | 7.2
SuperPoint + SuperGlue (ours) | 2.6 | 5.6 | 8.6
SP+SG w/ NOCS filtering ($d=0.03$) | 8.4 | 18.5 | 25.3
SP+SG w/ NOCS filtering ($d=0.05$) | 9.6 | 19.7 | 27.6
SP+SG w/ NOCS filtering ($d=0.1$) | 10.5 | 21.5 | 29.7
SP+SG w/ NOCS filtering ($d=0.2$) | 10.8 | 22.1 | 30.6
SP+SG w/ NOCS filtering ($d=0.5$) | 10.8 | 22.2 | 30.6
LoFTR [37] | 1.9 | 4.0 | 5.7
SP + LFM-3D (copied from Table 2) | 14.3 | 29.3 | 40.2
Table 3: Additional baselines for relative pose estimation on the _shoe_
Objectron sampled video frame pairs. We present results for relative pose
estimation in terms of Accuracy@$N$. At the bottom, we copy the LFM-3D results
from Table 2.

In this section, we compare our method to some other baseline methods for
relative pose estimation. Table 3 reports the same relative pose metrics as
Table 2 for four additional baselines:

  * •

Official SP+SG release \- We use the official SuperGlue weights provided by
the authors on github.

  * •

SP+SG w/ NOCS filtering \- We use the SuperPoint+SuperGlue (finetuned) model
to propose candidates, and only keep correspondences whose NOCS map values are
within distance $d$ of one another.

  * •

LoFTR [37] \- Dense feature matchers offer an alternative solution for
matching pairs with small co-visibility regions. We use the highly popular
LoFTR model (trained on outdoor scenes) provided by the Kornia [13] library to
propose correspondences between all image pair pixels and estimate the
fundamental matrix.

LFM-3D outperforms all of the additional baselines for the three challenging
Acc@N error thresholds.

#### Comparison to Official SuperGlue baseline.

Note that these results (Official SP+SG) are roughly equivalent to our own
SuperPoint+SuperGlue model trained on Megadepth, without finetuning (i.e.
SuperPoint + SuperGlue (ours)). This indicates that we have reproduced
SuperGlue in Tensorflow successfully.

#### Comparison to NOCS-based heuristics baseline.

Rather than integrating NOCS 3D signals in a learned manner, we experimented
with using a heuristics-based approach for filtering correspondences proposed
by the 2D-only method by NOCS distance (SP + SG w/ NOCS filtering). This naive
method of integrating NOCS coordinates underperforms LFM-3D, predictably. We
note that LFM-3D’s greatest improvements are in increased recall, as shown by
Figure 4. This filtering method cannot introduce new correspondences to the
limited set produced by the 2D-only baseline model.

#### Comparison to dense learnable matcher baseline.

The general-purpose dense matcher LoFTR [37] underperforms LFM-3D
significantly for the relative pose estimation task. We argue that this is due
to significant noise in the image background, and due to a relatively small
co-visibility region in which the dense learnable matcher can gain hints from.

##  Appendix C Ablation on number of keypoints

| _Shoe_
---|---
Method |  Acc@$5^{\circ}$ |  Acc@$10^{\circ}$ |  Acc@$15^{\circ}$
SP (1024) + SG | 11.0 | 22.2 | 31.6
SP (2048) + SG | 11.4 | 22.6 | 31.3
SP (4096) + SG | 10.7 | 21.6 | 30.6
SP (1024) + LFM-3D | 14.3 | 29.3 | 40.2
SP (2048) + LFM-3D | 15.5 | 30.6 | 42.3
SP (4096) + LFM-3D | 13.9 | 28.7 | 39.6
Table 4: Ablation study over number of extracted keypoints. We compare the
finetuned 2D-only baseline (SuperPoint [9] \+ SuperGlue [34]) to our LFM-3D
model from NOCS estimates on the _shoe_ Objectron evaluation set.

Table 4 compares LFM-3D with the finetuned 2D-only baseline with varying
numbers of keypoints (2048 and 4096, vs baseline 1024 from the original paper)
during relative pose estimation on Objectron _shoe_ image pairs. Note that we
do not retrain models for each new keypoint input size. Our results show that
LFM-3D benefits from additional keypoints more than the 2D-only baselines for
2048 keypoints, but performance drops with 4096 keypoints. We hypothesize that
our method can produce better features due to augmented 3D inputs, while in
the 2D-only case, having more features can introduce noise and false matches.

##  Appendix D Additional qualitative examples

We present additional qualitative results on the Objectron dataset in Figure
7. These results highlight that LFM-3D performs well across a variety of
shapes, textures, and lighting conditions.

Figure 7: Additional qualitative examples from sampled Objectron frames. For
each triplet, we compare SP+SG-MegaDepth, SP+SG-Finetuned, and LFM-3D.

## References

  * [1] Artsiom Ablavatski Jianing Wei Matthias Grundmann Adel Ahmadyan, Liangkai Zhang.  Objectron: A large scale dataset of object-centric videos in the wild with pose annotations.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [2] A. Barroso-Laguna, E. Riba, D. Ponsa, and K. Mikolajczyk.  Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters.  In Proc. ICCV, 2019.
  * [3] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool.  Speeded-Up Robust Features (SURF).  CVIU, 2008.
  * [4] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, and P. Fua.  BRIEF: Computing a Local Binary Descriptor Very Fast.  TPAMI, 2012.
  * [5] V. Chandrasekhar, G. Takacs, D. Chen, S. Tsai, R. Grzeszczuk, and B. Girod.  CHoG: Compressed histogram of gradients A low bit-rate feature descriptor.  In Proc. CVPR, 2009.
  * [6] H. Chen, Z. Luo, J. Zhang, L. Zhou, X. Bai, Z. Hu, C.-L. Tai, and L. Quan.  Learning to Match Features with Seeded Graph Matching Network.  In Proc. ICCV, 2021.
  * [7] H. Chen, Z. Luo, L. Zhou, Y. Tian, M. Zhen, T. Fang, D. McKinnon, Y. Tsin, and L. Quan.  ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer.  In Proc. ECCV, 2022.
  * [8] Javier Civera, Andrew J Davison, and JM Martinez Montiel.  Inverse depth parametrization for monocular slam.  IEEE transactions on robotics, 24(5):932–945, 2008.
  * [9] D. DeTone, T. Malisiewicz, and A. Rabinovich.  SuperPoint: Self-Supervised Interest Point Detection and Description.  In Proc. CVPR Workshops, 2018.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [11] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  arXiv preprint arXiv:2204.11918, 2022.
  * [12] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  In Proc. CVPR, 2019.
  * [13] D. Ponsa E. Rublee E. Riba, D. Mishkin and G. Bradski.  Kornia: an open source differentiable computer vision library for pytorch.  In Winter Conference on Applications of Computer Vision, 2020.
  * [14] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [15] M. Fischler and R. Bolles.  Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography.  Communications of the ACM, 1981.
  * [16] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 270–279, 2017.
  * [17] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297–7306, 2018.
  * [18] B. Haralick, C.-N. Lee, K. Ottenberga, and M. Nolle.  Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem.  IJCV, 1994.
  * [19] L. Kneip, D. Scaramuzza, and R. Siegwart.  A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation.  In Proc. CVPR, 2011.
  * [20] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2041–2050, 2018.
  * [21] D. Lowe.  Distinctive Image Features from Scale-Invariant Keypoints.  IJCV, 2004.
  * [22] Zixin Luo, Tianwei Shen, Lei Zhou, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan.  ContextDesc: Local Descriptor Augmentation with Cross-Modality Context.  In Proc. CVPR, 2019.
  * [23] W.-C. Ma, A. Yang, S. Wang, R. Urtasun, and A. Torralba.  Virtual Correspondence: Humans as a Cue for Extreme-View Geometry.  In Proc. CVPR, 2022.
  * [24] K. Mikolajczyk and C. Schmid.  Scale & Affine Invariant Interest Point Detectors.  IJCV, 2004.
  * [25] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I, pages 405–421, 2020.
  * [26] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas.  Working Hard to Know your Neighbor’s Margins: Local Descriptor Learning Loss.  In Proc. NIPS, 2017.
  * [27] D. Mishkin, F. Radenovic, and J. Matas.  Repeatability Is Not Enough: Learning Affine Regions via Discriminability.  In Proc. ECCV, 2018.
  * [28] H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han.  Large-Scale Image Retrieval with Attentive Deep Local Features.  In Proc. ICCV, 2017.
  * [29] Y. Ono, E. Trulls, P. Fua, and K. Moo Yi.  LF-Net: Learning Local Features from Images.  In Proc. NIPS, 2018.
  * [30] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12179–12188, 2021.
  * [31] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
  * [32] Jerome Revaud, Cesar De Souze, Philippe Weinzaepfel, and Martin Humenberger.  R2D2: Repeatable and Reliable Detector and Descriptor.  In Proc. NeurIPS, 2019.
  * [33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [34] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich.  SuperGlue: Learning Feature Matching with Graph Neural Networks.  In Proc. CVPR, 2020.
  * [35] T. Sattler, B. Leibe, and L. Kobbelt.  SCRAMSAC: Improving RANSAC’s efficiency with a spatial consistency filter.  In Proc. ICCV, 2009.
  * [36] E. Simo-Serra, E. Trulls, L. Ferras, I. Kokkinos, P. Fua, and F. Moreno-Noguer.  Discriminative Learning of Deep Convolutional Feature Point Descriptors.  In Proc. ICCV, 2015.
  * [37] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou.  LoFTR: Detector-Free Local Feature Matching with Transformers.  In Proc. CVPR, 2021.
  * [38] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [39] Y. Tian, B. Fan, and F. Wu.  L2-Net: Deep Learning of Discriminative Patch Descriptor in Euclidean Space.  In Proc. CVPR, 2017.
  * [40] Y. Tian, X. Yu, B. Fan, F. Wu, H. Heijnen, and V. Balntas.  SOSNet: Second Order Similarity Regularization for Local Descriptor Learning.  In Proc. CVPR, 2019.
  * [41] C. Toft, D. Turmukhambetov, T. Sattler, F. Kahl, and G. Brostow.  Single-Image Depth Prediction Makes Feature Matching Easier.  In Proc. ECCV, 2020.
  * [42] E. Tola, V. Lepetit, and P. Fua.  DAISY: An Efficient Dense Descriptor Applied to Wide-Baseline Stereo.  TPAMI, 2010.
  * [43] T. Tuytelaars and L. Van Gool.  Wide baseline stereo matching based on local, affinely invariant regions.  In Proc. BMVC, 2000.
  * [44] M. Tyszkiewicz, P. Fua, and E. Trulls.  DISK: Learning local features with policy gradient.  In Proc. NeurIPS, 2020.
  * [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [46] Yannick Verdie, Kwang Moo Yi, Pascal Fua, and Vincent Lepetit.  TILDE: A Temporally Invariant Learned Detector.  In Proc. CVPR, 2015.
  * [47] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. Guibas.  Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation.  In Proc. CVPR, 2019.
  * [48] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely.  Learning feature descriptors using camera pose supervision.  In Proc. European Conference on Computer Vision (ECCV), 2020.
  * [49] K. M. Yi, E. Trulls, V. Lepetit, and P. Fua.  LIFT: Learned Invariant Feature Transform.  In Proc. ECCV, 2016.
  * [50] K. M. Yi, E. Trulls, Y. Ono, V. Lepetit, M. Salzmann, and P. Fua.  Learning to Find Good Correspondences.  In Proc. CVPR, 2018.
  * [51] Kwang Moo Yi, Yannick Verdie, Pascal Fua, and Vincent Lepetit.  Learning to Assign Orientations to Feature Points.  In Proc. CVPR, 2016.
  * [52] J. Zhang, D. Sun, Z. Luo, A. Yao, L. Zhou, T. Shen, Y. Chen, L. Quan, and H. Liao.  Learning two-view correspondences and geometry using order-aware network.  In Proc. ICCV, 2019.