# iMatching: Imperative Correspondence Learning

Zitong Zhan1111Equal contribution., Dasong Gao2111Equal contribution., Yun-Jou
Lin3, Youjie Xia3, Chen Wang1222Corresponding author.
1SAIR Lab, IAD, CSE, University at Buffalo, 2MIT, 3InnoPeak Technology
{zitongz<EMAIL_ADDRESS><EMAIL_ADDRESS>{rose.lin<EMAIL_ADDRESS>
###### Abstract

Learning feature correspondence is a foundational task in computer vision,
holding immense importance for downstream applications such as visual odometry
and 3D reconstruction. Despite recent progress in data-driven models, feature
correspondence learning is still limited by the lack of accurate per-pixel
correspondence labels. To overcome this difficulty, we introduce a new self-
supervised scheme, imperative learning (IL), for training feature
correspondence. It enables correspondence learning on arbitrary uninterrupted
videos without any camera pose or depth labels, heralding a new era for self-
supervised correspondence learning. Specifically, we formulated the problem of
correspondence learning as a bilevel optimization, which takes the
reprojection error from bundle adjustment as a supervisory signal for the
model. To avoid large memory and computation overhead, we leverage the
stationary point to effectively back-propagate the implicit gradients through
bundle adjustment. Through extensive experiments, we demonstrate superior
performance on tasks including feature matching and pose estimation, in which
we obtained an average of 30% accuracy gain over the state-of-the-art matching
models. The source code will be made public to benefit the community.

Figure 1: iMatching learns feature correspondence from a bundle adjustment
(right), eliminating the need of any ground truth label. Bundle adjustment
finds the best landmark position and camera pose and can be used as self-
supervision signal for feature correspondence networks.

##  1 Introduction

Feature correspondence has long been recognized as a pivotal task in computer
vision, acting as a cornerstone for many critical downstream applications such
as visual odometry [35], 3D reconstruction [27], and object identification
[11]. The essence of accurately matching features across diverse images and
viewpoints is invaluable in constructing coherent and precise representations
of the visual world.

While recent years have witnessed significant advancements in data-driven
models, the task remains intrinsically challenging [21, 1, 28], predominantly
due to the absence of sufficient pixel-level annotated data for point
matching. Unlike the labels for other computer vision tasks such as object
detection where ground truth annotations can be drawn by hand, annotating
feature correspondence for every pixel is almost impossible. This constraint
has inadvertently limited the potential and generalization ability of the
state-of-the-art data-driven models [24, 18]. Moreover, the domain gap in
visual appearances between training and testing data also greatly discounted
model performance [13].

To alleviate the data problem, there are three common ways. The first way is
to simulate correspondence annotation, either from homography transformation
[3] or synthetic environments [33]. However, this method often falls short in
capturing real-world variations like changes in lighting and occlusions,
leading to the sim-to-real gap. The second way is to leverage physical sensors
to capture ground truth such as camera pose or depth information. However,
this requires specialized equipment like high-end motion capture systems,
usually only available indoors and infeasible for outdoor scenes or unique
settings like underground environments [40]. Moreover, sensors such as depth
cameras and LiDAR tend to offer inaccurate and incomplete depth measurements
[16], which are inadequate for pixel-level matching [24].

A third way to address the challenge of obtaining large, diverse, and
accurately labeled image correspondence datasets has been weakly- and self-
supervised methodologies. For example, SGP [37] proposes a student-teacher
framework for training the CAPS correspondence model [32] in a self-supervised
manner. In SGP, the teacher (RANSAC) and the student model are updated
alternately, allowing the teacher to produce pseudo-pose labels for the
student model. Nonetheless, this design is limited to models like CAPS, which
uses epipolar- instead of reprojection-based loss during training. Moreover,
the teacher and student model training is isolated, leading to potential error
propagation, _i.e_., mistakes made by the teacher model can be propagated to
the student model, potentially amplifying the issues.

To fully address the data problem, we explore the third category and propose a
new self-supervised end-to-end learning framework, iMatching (Figure 1), for
feature correspondence using an emerging technique, imperative learning [36,
6]. For the first time, we formulate the problem of feature matching as a
bilevel optimization [9] where we direct the learning of the model parameters
with an optimization procedure, _i.e_., bundle adjustment (BA), which is
itself an optimization process. Specifically, at the lower level (BA), we
optimize the camera pose and the 3D landmark locations that best explain the
model-predicted correspondences, whereas, at the higher level (gradient
descent), we optimize the network parameters such that the correspondence
prediction results in more consistent poses and landmark locations.

Our new formulation for feature correspondence based on bilevel optimization
brings distinctive benefits: The low-level optimization eliminates the need
for ground truth geometric labels, thanks to the self-correction mechanism
brought by bundle adjustment. Such a design can be generalized to any up-to-
date matching model. In the experiments, we show that our method is plug-and-
play and significantly boosts the performance of state-of-the-art (SOTA)
models [5, 28, 1, 14] by an average of 30% and a maximum 82% on pixel-level
feature matching on unseen, image-only datasets.

One of the challenges of our formulation is model training which requires
differentiating the network through low-level optimization (BA). A popular
solution is to backpropagate the errors through the unrolled iterative
optimization [8, 30], which is extremely resource-demanding as the temporary
variables generated during each iteration need to be stored and traversed for
gradient calculation. Instead, we specially designed our loss function so that
we can bypass the iterative process during backward pass while still obtaining
the gradient. This allowed us to employ a black-box, non-differentiable bundle
adjustment to ensure the robustness of training while learning end-to-end as
if it were differentiable. This end-to-end learnable design also enables the
possibility of performing online learning. In summary, our contributions are
summarized as follows:

  * •

We propose the first imperative learning (IL)-based framework for training
feature correspondence models with uninterrupted videos without any labels
such as pose or depth.

  * •

We design an efficient method for propagating the gradient through the bundle
adjustment so that we can solve the optimization without differentiating
individual steps of the optimization, making end-to-end training possible.

  * •

Through extensive experiments and evaluation on a variety of tasks, we show
the superiority of the proposed framework by boosting the performance of the
SOTA feature-matching model by 30% precision on the pose estimation task and
13.6% on the feature matching task.

##  2 Related Work

Data-driven methods for feature matching categorized as supervised, weakly- or
self-supervised will be introduced in Sec. 2.1 and 2.2, respectively.
Additionally, we summarize imperative learning and data-driven models
involving bundle adjustment in Sec. 2.3 and 2.4, respectively.

###  2.1 Supervised Methods

Supervised methods rely on known per-pixel correspondences obtained either
from (1) synthetic data or (2) indirect ground truth such as depth images and
camera poses collected from physical sensors. The first category uses photo-
realistic renderers [33] or data augmentation techniques to generate training
images. For example, LIFT [38] learns from image samples capturing the same
view but with strong illumination change. SuperPoint [3] relies on generic
image datasets with homographic transformations. However, those synthetic data
techniques may not generalize well to the real-world scenario. For example,
homographic transformations can only model scenes where all the points are
coplanar.

Most of the existing methods [21, 31, 4, 19, 28, 1, 5] belong to the second
category, which leverage collected ground truth depth scan and camera pose
data from physical sensors such as motion capture system (MoCap), depth
camera, or LiDAR. However, MoCap is confined to dedicated working zones.
Scenes like underground urban sewer systems or mines [10] are unable to
accommodate complex MoCap devices; depth cameras are limited to indoor
environments [16] and provide limited precision, and LiDAR sensors can only
provide sparse points. As a result, 3D scene data have a significantly smaller
scale compared with generic 2D image due to the complexity of data collection.
3D real world datasets choices are limited. There are only a few datasets such
as MegaDepth [15] and ScanNet [2] available for feature correspondence
learning, but those datasets are limited to small-scale environments or lack
data diversity [24].

###  2.2 Weakly- and Self-supervised Methods

To address the challenges of obtaining large, diverse image datasets with
accurate correspondence labels, several efforts have focused on weakly-
supervised methodologies. For instance, CAPS [32] and Patch2Pix [41] eliminate
the need for depth information and depend solely on relative camera poses.
They leverage the differentiability of matchers by penalizing both epipolar
distance and cycle inconsistency.

To further remove the dependency on camera poses, SGP [37] introduced a self-
supervised method for training epipolor geometry-based models like CAPS [32]
using the teacher-student framework. It introduces RANSAC pose estimation as a
teacher to generate pseudo labels for training the student CAPS model.
However, the student and teacher training are isolated, which may lead to
potential error propagation. Different from SGP, our proposed self-supervised
learning framework formulated as bilevel optimization is end-to-end and
workable for all models that are directly supervised by pixel locations. This
self-supervised dense learning framework can be applied to a wide variety of
models as demonstrated in the experiments.

###  2.3 Imperative Learning

Imperative learning (IL) is an emerging self-supervised learning framework
based on bilevel optimization. It has been used to tackle problems where both
data-driven models and geometry problems are involved. For example, iPlanner
[36] adopted a B-spline interpolation [39] as the low-level optimization to
guide the network to generate smoothed trajectories for path planning; whereas
iSLAM [6] utilized the pose graph optimization to achieve the reciprocal
learning for the front-end and back-end in a simultaneous location and mapping
(SLAM) system. To the best of our knowledge, IL has not been applied to the
problem of feature correspondence learning, and iMatching is the first method
for self-supervised feature matching using bundle adjustment.

###  2.4 Bundle Adjustment in Deep Learning

Bundle adjustment (BA) is an optimization technique used in computer vision
and photogrammetry to simultaneously estimate the 3D coordinates of landmarks
and camera parameters by minimizing the reprojection errors between observed
and predicted image points. Recently, several works have tried to integrate BA
as part of a neural network. For example, DROID-SLAM [30] uses BA to correct
single-view depth estimation during inference so that it is consistent with
the geometry shape. For training dense depth and camera poses, BA can be used
to encourage geometric consistency between pose and depth prediction [26].
However, to make the BA process differentiable, existing methods [30, 26, 8]
unroll the entire iterative second-order optimization process as part of the
auto-differentiation graph. This incurs a significant computation and memory
overhead since the computation graphs of each iteration need to be kept in
memory and traversed during backpropagation. In contrast, we leverage the
stationary point of BA, resulting in an extremely efficient one-step implicit
gradient computation. As an additional benefit, such a formulation allows us
to freely leverage non-differentiable techniques such as alternating pruning
and refinement (Sec. 3.1.2) to enhance the robustness of BA.

##  3 Imperative Correspondence Learning

We present an overview of iMatching training scheme and its formulation in
Sec. 3.1. The matching network and design of BA are presented in Sec. 3.1.1
and 3.1.2, respectively.

Figure 2: The iMatching framework for feature correspondence learning consists
of a feature correspondence network and a bundle adjustment process, which
results in a bilevel optimization.

###  3.1 Problem Formulation

The architecture of our iMatching training scheme is shown in Figure 2, which
can be divided into two parts, a feature correspondence network
$f_{\bm{\theta}}$ parameterized by $\bm{\theta}$ and a BA optimization process
represented by (1b). The system is formulated as a bilevel optimization
problem, expressed as:

$\displaystyle\min_{\bm{\theta}}$
$\displaystyle\;\;\mathcal{R}(f_{\bm{\theta}};\mathbf{T}^{*},\mathbf{p}^{*}),$
(1a) $\displaystyle\operatorname{s.t.}$
$\displaystyle\;\;\mathbf{T}^{*},\mathbf{p}^{*}=\operatorname*{argmin}_{{\mathbf{T},\mathbf{p}}}\mathcal{R}(\mathbf{T},\mathbf{p};f_{\bm{\theta}}),$
(1b)

where $\mathcal{R}$ is the reprojection error function;
$\mathbf{T}_{t}\in\mathbb{SE}(3)$ are the camera poses in Lie group
representation; and $\mathbf{p}_{i}\in\mathbb{R}^{3}$ are the 3D landmark
positions. Intuitively, at the lower level, BA finds _the most likely poses
and landmark positions_ given feature correspondence inferred using network
parameters $\bm{\theta}$; whereas at the upper level, it optimizes for _the
best parameters_ $\bm{\theta}$ for least reprojection error $\mathcal{R}$
given the estimated poses and landmark positions from the lower level. This
training scheme is named imperative learning due to this passive reciprocal
optimization between the two levels. Since both levels only involve minimizing
the projection error, it eliminates the requirements in any form of ground
truth. We next present the details of the correspondence network and BA with
the solution to bilevel optimization detailed in Sec. 3.2.

####  3.1.1 Feature Correspondence

Given image pairs $(I_{i},I_{j})$, the feature correspondence network
$f_{\bm{\theta}}$ produces a set of $K$ point correspondences:

$f_{\bm{\theta}}(I_{i},I_{j})\mathrel{\mathop{\mathchar
58\relax}}=\\{(\textbf{u}_{i}^{(k)},\textbf{u}_{j}^{(k)})\\}_{k=1}^{K},$ (2)

where $\textbf{u}_{i}^{(k)}$ and $\textbf{u}_{j}^{(k)}$ are the 2D coordinates
on image $I_{i}$ and $I_{j}$, respectively. In practice, given a batch of $B$
images, we match each image with its past $n$ $(1<n<B)$ frames.

####  3.1.2 Bundle Adjustment

Given a set of correspondences, the BA process seeks to find the optimal pose
of the cameras $\mathbf{T^{*}}=\\{T_{i}\\}_{i=1}^{B}$ and the 3D positions of
landmarks $\mathbf{p^{*}}=\\{p_{l}\\}_{l=1}^{N}$ that minimizes the overall
reprojection error $\mathcal{R}$ defined as:

$\displaystyle\mathcal{R}(\mathbf{T},\mathbf{p};f_{\bm{\theta}})=\sum_{j=1}^{B}\sum_{i=j-n}^{j-1}\sum_{l=1}^{P}\norm{\mathbf{u}_{i}^{(k)}-\pi\left(p_{l};T_{i}\right)}_{2}^{2},$
(3)

with $\pi$ being the reprojection function and $l$ being the index of the
point $p_{l}$, which is triangulated from $\textbf{u}_{i}^{(k)}$ and
$\textbf{u}_{j}^{(k)}$.

Next, we describe our design choices that make the BA reliable. As the sole
source of supervision, the robustness of BA is fundamental to the success of
the training. The BA can be roughly divided into initialization and pruning
stages. To obtain an accurate initialization of poses and landmark locations
for all frames in the minibatch, we bootstrap the map by localizing and
triangulating on a subset of frames before finding the pose of all frames.
After all frames are properly initialized, we run a multi-pass BA scheme that
alternates between optimization and outlier rejection. We describe each of the
steps in the rest of this section.

Figure 3: Iterative bundle adjustment first initializes the camera poses and
landmark positions, and then alternates between optimizing reprojection error
and outlier rejection until convergence.

##### Map Initialization

Assume each batched sample $\\{I_{t}\\}_{t=1}^{B}$ has $B$ frames from an
input sequence. To ensure the robustness and performance of BA, we run
structure-from-motion (SfM) on the first $d$ frames ($d<B$). We first find a
frame pair with the most correspondences and parallax as the anchor, and then
triangulate the positions of the landmarks within the pair. For other frames,
we estimate their transformations with respect to the anchor frames by solving
an EPnP [12] on given triangulated 3D positions and 2D correspondences.
Lastly, with each frame in the SfM subset having a known camera pose, we
triangulate all unused correspondences to obtain the maximum number of
landmarks.

##### Frame Tracking

For the remaining $B-d$ frames, we predict their poses sequentially using a
constant-velocity motion model for a rough estimation and then refine the pose
estimations by performing a 2D-3D registration against the landmark map
generated in the previous step. Existing landmarks observed by those frames
are refined by a RANSAC-based multiview triangulation [7].

##### Iterative Bundle Adjustment

In this step, we utilize the BA to refine the landmarks and poses estimated
above. In practice, single-pass BA is found to be prone to poor convergence
due to noise-corrupted keypoint detection and wrong feature correspondences
[34, 23]. To solve this challenge, we follow a procedure similar to that of
COLMAP [22] that alternates joint optimization and outlier rejection.
Specifically, after optimizing the BA objective (1b) with the Levenberg-
Marquardt (LM) optimizer, each landmark goes through a filter and re-
triangulation process: 3D locations with large reprojection errors are pruned
from the set of landmarks at first, and then the surviving ones are updated by
re-triangulated landmark locations with the refined camera pose and 2D
correspondences. This alternative procedure runs for a small number of
iterations. The final poses and landmark positions are used to train the
feature correspondence network.

###  3.2 Optimization

To train the network $f_{\bm{\theta}}$ via gradient descent
$\bm{\theta}\leftarrow\bm{\theta}-\eta\partialderivative{\mathcal{R}}{\bm{\theta}}$,
we need to evaluate the derivative of the reprojection error in (1a) w.r.t
parameters $\bm{\theta}$ of the matching network. According to the chain rule,
we can calculate the derivative $\partialderivative{\mathcal{R}}{\bm{\theta}}$
as

$\partialderivative{\mathcal{R}}{\bm{\theta}}=\left(\partialderivative{\mathcal{R}}{\mathbf{T}^{*}}\partialderivative{\mathbf{T}^{*}}{f_{\bm{\theta}}}+\partialderivative{\mathcal{R}}{\mathbf{{p}}^{*}}\partialderivative{\mathbf{{p}}^{*}}{f_{\bm{\theta}}}+\partialderivative{\mathcal{R}}{f_{\bm{\theta}}}\right)\partialderivative{f_{\bm{\theta}}}{\bm{\theta}}.$
(4)

However, Equation (4) is well defined only if two conditions are satisfied:
(1) the matching network $f_{\bm{\theta}}$ is differentiable, _i.e_.,
$\partialderivative{f_{\bm{\theta}}}{\theta}$ exists; and (2) the BA in the
low-level optimization (1b) is differentiable, _i.e_.,
$\partialderivative{\mathbf{{T}}^{*}}{f_{\bm{\theta}}}$ and
$\partialderivative{\mathbf{{x}}^{*}}{f_{\bm{\theta}}}$ exist. However, the
challenges of satisfying the two conditions are (1) feature matching is
normally represented by matching of discrete pixel locations, while
differentiating through the integer pixel locations is unstraightforward; (2)
solving the bundle adjustment requires iterative, 2-order optimizers like LM,
which computes large Jacobian matrices and its inversion. This already
computationally heavy process makes back-propagating through it difficult. We
next describe our solutions to resolve the two challenges.

####  3.2.1 Differentiable Image Correspondences

A feature correspondence model $f_{\bm{\theta}}$ typically consists of three
stages: keypoint detection (optional), feature extraction, and matching
prediction [14]. Feature descriptors are generally derived from feature maps
$F_{i}$ produced by standard backbone networks, through interpolation at the
locations of keypoints. The process of matching prediction involves
determining the target feature locations based on extracted descriptors. To
make the predicted correspondences differentiable, current methods typically
compute the feature locations by expectation [32, 41] or regression [5, 1].

##### Expectation-based Matching Prediction

Given a feature coordinate $\mathbf{u}_{i}^{(k)}$ from $I_{i}$ and feature
maps $F_{i},F_{j}$, the matching coordinate on image $j$ is an expectation
over all possible target feature locations weighted by feature similarity

$\mathbf{u}_{j}^{(k)}=\frac{1}{HW}\sum_{\mathbf{u}^{(l)}\in
I_{j}}\mathbf{u}^{(l)}\cdot
S\left(F_{i}(\mathbf{u}_{i}^{(k)}),F_{j}(\mathbf{u}^{(l)})\right),$ (5)

where $S$ can be cosine similarity as an example.

##### Regression-based Matching Prediction

As a more generalized version of expectation-based prediction, techniques in
this category directly regress the amount of feature shift from the feature
maps $F_{i}$ and $F_{j}$, using a neural network:

$\mathbf{u}_{j}^{(k)}=g_{\bm{\theta}^{\prime}}\left(F_{i},F_{j};\mathbf{u}_{i}^{(k)}\right),$
(6)

where $g_{\bm{\theta}^{\prime}}$ can be a transformer [1] or a CNN [5]. It is
worth noting that iMatching can be used for both categories.

However, there are two non-differentiable cases. First, some keypoint
detection networks like SuperPoint [3] have a ranking process with a discrete
nature: feature points $\\{\mathbf{u}_{i}^{(k)}\\}_{k=1}^{K}$ are typically
picked from the pixel coordinates with top $K$ confidence scores. To handle
such cases, we can either replace the detection network by a differentiable
one or freeze its parameters and thus need not propagate the gradient through
it. Second, some methods like [3, 21] use search-based matching prediction.
This is completely non-differentiable and is out of the scope of our
discussion.

####  3.2.2 Differentiable Bundle Adjustment

Figure 4: Gradient flow in iMatching. During training, the gradient from the
reprojection and epipolar losses is only backpropagated through the 2D feature
correspondences $\mathbf{u}_{j}^{(k)}$ to train the feature matching network.
According to our analysis, backpropagation through the robust iterative
estimation could be avoided due to the stationarity of optimization outputs
$\mathbf{{p}}$ and $\mathbf{T}$.

An existing widely-used approach to have a differentiable bundle adjustment is
to unroll the optimization loop for a fixed number of steps and treat them as
part of the forward pass, as done by [8, 30, 29]. However, such a strategy
requires the retention of each iteration’s computation graph and the ability
to compute the higher-order gradient, which makes this method inapplicable to
large-scale problems.

To avoid explicitly computing
$\partialderivative{\mathbf{{T}}^{*}}{f_{\bm{\theta}}}$ and
$\partialderivative{\mathbf{{p}}^{*}}{f_{\bm{\theta}}}$ by differentiating
through the iterative optimization process, we leverage optimization of BA at
convergence, which allows us to efficiently backpropagate gradient through the
lower level optimization without compromising its robustness. Concretely,
since the optimization of BA is unconstrained, at its stationary point (either
locally or globally), we have
$\partialderivative{\mathcal{R}}{\mathbf{T}^{*}}\approx\mathbf{0}$,
$\partialderivative{\mathcal{R}}{\mathbf{{p^{*}}}}\approx\mathbf{0}$. Then the
gradient (4) becomes

$\partialderivative{\mathcal{R}}{\bm{\theta}}=\partialderivative{\mathcal{R}}{f_{\bm{\theta}}}\partialderivative{f_{\bm{\theta}}}{\bm{\theta}}.$
(7)

This implies that to obtain the gradient descent update (1b), it is sufficient
to evaluate the reprojection error only once after bundle adjustment and back-
propagate through the correspondence argument, treating $\mathbf{{T}^{*}}$ and
$\mathbf{{p}^{*}}$ as given. Then, $f_{\bm{\theta}}$ can be trained with SGD
with the gradient given by (7). Figure 4 illustrates such a gradient flow.

It is worth noting that Ji et al. [9, Table 1, N-loop AID] proved that, as
long as the upper-level optimization has a properly small step size, the
bilevel optimization is guaranteed to converge, even if the lower-level
optimization has not converged. Although [9] assumes that the lower-level
optimization uses 1-order optimizers like SGD, we empirically found the
bilevel optimization can still converge using 2-order optimizers with just a
few iterations.

####  3.2.3 Improving Training Convergence

To further improve the convergence of training, we retain the coarse-to-fine
training scheme which is common in feature correspondence models. These models
typically first obtain a rough estimate having the same format as in Sec.
3.1.1 from low-resolution image features, and then refine the result using a
higher-resolution feature or a cropped image patch to produce fine matching.
While the BA in Sec. 3.1.2 is only applied on the fine estimate for the best
accuracy, the initial rough estimate receives the same gradient update in (7).

Figure 5: Pose estimation comparison between pretrained and finetuned model.
The bottom point clouds are reconstructed by aligning RGBD scans using RANSAC-
estimated poses derived from feature correspondence. The pretrained CAPS model
(left) creates artifacts on the sofa and monitor due to inaccurately estimated
poses and falsely aligned depth map, while such artifacts are not presented in
the finetuned model (right). Best viewed digitally.

Scene | Indoor | Outdoor | Natural | Artificial | Mixed | Easy | Hard | Overall
---|---|---|---|---|---|---|---|---
reproj thres | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px | 1px | 2px | 5px
(a) Non-differentiable Matcher
SIFT | 22.6 | 31.7 | 41.9 | 22.9 | 31.0 | 38.3 | 16.0 | 23.2 | 30.5 | 24.9 | 33.8 | 42.7 | 24.1 | 32.5 | 39.7 | 23.7 | 32.5 | 40.8 | 21.4 | 29.4 | 37.4 | 22.8 | 31.2 | 39.3
ORB | 20.8 | 40.2 | 58.5 | 16.5 | 32.0 | 45.9 | 10.2 | 21.2 | 33.0 | 20.1 | 38.4 | 55.0 | 18.5 | 35.9 | 50.9 | 18.8 | 36.5 | 52.1 | 16.7 | 32.4 | 47.2 | 17.7 | 34.4 | 49.6
SuperGlue | 39.6 | 64.2 | 85.0 | 38.2 | 62.4 | 80.9 | 27.2 | 49.0 | 71.0 | 41.9 | 66.8 | 85.6 | 40.4 | 65.4 | 83.2 | 40.7 | 65.8 | 84.1 | 36.7 | 60.2 | 80.2 | 38.6 | 62.9 | 82.1
R2D2 | 30.6 | 48.6 | 64.8 | 31.4 | 48.6 | 61.0 | 25.2 | 41.4 | 55.1 | 32.9 | 50.5 | 64.7 | 32.0 | 50.1 | 62.4 | 32.3 | 50.6 | 64.3 | 29.1 | 45.7 | 59.6 | 31.1 | 48.6 | 62.2
(b) Pretrained Models of End-to-end Differentiable Matcher (Supervised)
CAPS | 26.6 | 62.3 | 87.3 | 27.1 | 63.3 | 87.3 | 21.1 | 52.7 | 81.0 | 28.0 | 64.6 | 88.1 | 29.0 | 67.1 | 90.0 | 28.4 | 66.2 | 89.4 | 25.5 | 60.1 | 85.3 | 26.9 | 63.0 | 87.3
Patch2Pix | 0.7 | 3.5 | 23.6 | 0.9 | 4.6 | 25.8 | 1.0 | 4.9 | 26.1 | 0.8 | 4.1 | 24.9 | 0.8 | 4.3 | 25.0 | 0.9 | 4.6 | 26.6 | 0.8 | 4.0 | 23.9 | 0.8 | 4.3 | 25.2
ASpanFormer | 51.5 | 76.9 | 94.2 | 51.5 | 76.7 | 93.3 | 43.7 | 70.1 | 90.6 | 52.2 | 77.8 | 94.3 | 55.4 | 79.3 | 94.2 | 55.8 | 80.9 | 95.2 | 47.4 | 72.9 | 92.0 | 51.5 | 76.8 | 93.6
(c) Self-supervised Baseline
SGP | 44.8 | 70.5 | 88.7 | 47.7 | 71.8 | 88.7 | 36.6 | 60.7 | 83.1 | 47.5 | 72.6 | 89.2 | 52.5 | 76.5 | 91.6 | 50.8 | 75.5 | 90.8 | 43.1 | 67.6 | 86.7 | 46.8 | 71.4 | 88.7
(d) Models Using Our iMatching Training Scheme (Self-supervised)
iCAPS (Ours) | 47.5 | 70.9 | 88.3 | 49.7 | 72.9 | 89.7 | 40.4 | 63.8 | 85.6 | 50.1 | 73.3 | 89.4 | 53.0 | 76.3 | 91.5 | 53.0 | 76.0 | 91.0 | 45.3 | 68.9 | 87.7 | 49.0 | 72.3 | 89.3
iPatch2Pix (Ours) | 6.1 | 17.3 | 42.1 | 7.5 | 20.6 | 44.5 | 7.3 | 20.2 | 43.8 | 5.8 | 17.4 | 42.5 | 9.3 | 23.4 | 46.3 | 8.3 | 22.9 | 49.0 | 6.0 | 16.7 | 39.0 | 7.1 | 19.7 | 43.8
iASpan (Ours) | 60.1 | 79.0 |  95.0 | 57.8 | 78.7 | 94.2 | 51.7 | 73.6 |  91.8 | 58.6 | 78.9 |  95.1 | 62.7 | 82.0 | 94.9 | 63.1 | 82.9 |  96.0 | 54.0 | 74.9 | 92.9 | 58.4 | 78.8 | 94.4

Table 1: Mean matching accuracy (%) on the TartanAir dataset under error
tolerance of 1, 2, and 5 pixels. Our iMatching training scheme consistently
boosts the accuracy of feature matching models, with finetuned results shown
in “iCAPS", “iPatch2Pix", and “iASpan".

##  4 Experiments

We show the effectiveness of our method by presenting the performance gain
after training SOTA correspondence models with iMatching on unseen, _image-
only_ datasets. We experiment with CAPS [32], Patch2Pix [41], ASpanFormer [1],
and DKM [5], and name the models using our iMatching training scheme as iCAPS,
iPatch2Pix, iASpan, and iDKM respectively. We also compare with SGP [37],
which, to the best of our knowledge, is the only self-supervised method.

###  4.1 Datasets

##### TartanAir

[33] is a large (3TB) and diverse synthetic SLAM dataset. Collected in photo-
realistic simulation environments, the dataset contains precise ground truth
depth and pose labels and covers a wide range of scene types as well as
challenging conditions such as dynamic lighting and adverse weather. We use
TartanAir to evaluate pixel-level matching accuracy since it is synthesized
and thus free of sensor noises. We exclude scenes where ground truth matching
is unavailable, e.g., textureless areas or fast moving dynamic objects. We
divide each valid scene into subsets of size 8.5:0.5:1 for training,
validation, and testing, respectively.

##### ETH3D-SLAM

[25] provides RGBD sequences captured in the real world. Its scenes primarily
consist of small-scale objects with complex details. We exclude the scenes
where RGB camera do not provide information in a completely dark environment
or is moved while its view is blocked.

MegaDepth [15] and ScanNet [2] datasets are not employed because they only
provide discrete images, while iMatching assumes uninterrupted videos to be
available. Moreover, these datasets are not suitable for evaluating feature
matching because of a lack of pixel-level accuracy, as their ground truth is
generated from structure-from-motion.

Method |  SuperGlue [21] |  SGP [37] |  CAPS [32] | iCAPS (Ours) |  DKM [5] | iDKM (Ours)
---|---|---|---|---|---|---
AUC | 5° | 10° | 20° | 5° | 10° | 20° | 5° | 10° | 20° | 5° | 10° | 20° |  % $\uparrow$ | 5° | 10° | 20° | 5° | 10° | 20° |  % $\uparrow$
cables | 66.9 | 72.6 | 75.4 | 62.0 | 67.9 | 70.9 | 67.4 | 73.9 | 77.1 | 69.0 | 76.3 | 80.0 | 2.4% | 59.2 | 63.7 | 66.0 | 84.4 | 91.6 | 95.2 | 42.6%
camera_shake | 64.5 | 78.0 | 86.1 | 68.4 | 82.1 | 89.8 | 63.0 | 77.8 | 87.0 | 72.3 | 84.1 | 90.8 | 14.8% | 65.9 | 72.9 | 76.5 | 67.7 | 74.2 | 77.4 | 2.7%
ceiling | 81.0 | 86.6 | 89.7 | 78.9 | 85.0 | 88.1 | 81.4 | 87.8 | 91.1 | 83.9 | 91.0 | 94.6 | 3.1% | 59.4 | 62.8 | 64.6 | 76.6 | 83.8 | 87.5 | 29.0%
desk | 77.1 | 84.5 | 88.4 | 73.5 | 82.3 | 87.0 | 72.8 | 82.2 | 87.3 | 74.4 | 83.9 | 88.9 | 2.2% | 82.8 | 85.3 | 86.8 | 82.9 | 85.5 | 86.9 | 0.1%
desk_changing | 71.3 | 77.2 | 80.4 | 69.7 | 76.8 | 80.4 | 73.5 | 81.6 | 85.8 | 74.3 | 83.1 | 87.8 | 1.1% | 61.2 | 62.9 | 63.7 | 75.3 | 86.1 | 91.8 | 23.0%
einstein | 57.1 | 62.0 | 64.8 | 66.7 | 72.4 | 75.8 | 67.8 | 74.1 | 77.8 | 69.8 | 76.6 | 80.5 | 2.9% | 36.6 | 38.2 | 39.4 | 63.4 | 68.4 | 71.3 | 73.2%
einstein_GLC | 42.3 | 46.1 | 48.4 | 51.7 | 57.4 | 60.7 | 51.3 | 56.7 | 60.3 | 53.6 | 59.3 | 63.0 | 4.5% | 35.5 | 41.0 | 45.9 | 53.1 | 62.1 | 69.9 | 49.6%
mannequin | 76.2 | 80.6 | 83.0 | 80.1 | 84.9 | 87.5 | 80.2 | 85.3 | 88.0 | 83.9 | 89.5 | 92.5 | 4.6% | 59.5 | 60.9 | 61.6 | 81.1 | 88.1 | 91.6 | 36.3%
mannequin_face | 69.1 | 71.0 | 71.9 | 73.2 | 76.3 | 77.9 | 73.4 | 76.8 | 78.5 | 77.2 | 81.3 | 83.3 | 5.2% | 53.9 | 54.3 | 54.5 | 71.3 | 73.7 | 74.8 | 32.3%
planar | 67.7 | 78.9 | 84.5 | 65.6 | 79.5 | 86.7 | 67.1 | 81.6 | 88.9 | 69.3 | 83.3 | 91.0 | 3.3% | 62.9 | 70.6 | 74.4 | 71.3 | 81.1 | 86.4 | 13.4%
plant | 78.5 | 82.2 | 84.1 | 74.4 | 80.2 | 83.0 | 71.9 | 78.5 | 81.7 | 81.0 | 88.0 | 91.5 | 12.7% | 86.5 | 88.9 | 90.0 | 88.4 | 91.1 | 92.4 | 2.2%
plant_scene | 72.6 | 77.2 | 79.6 | 71.0 | 76.4 | 79.2 | 71.8 | 77.7 | 80.7 | 80.2 | 86.8 | 90.2 | 11.7% | 44.2 | 45.2 | 45.7 | 83.5 | 90.7 | 94.3 | 88.9%
sfm_lab_room | 87.7 | 93.8 | 96.9 | 90.6 | 95.3 |  97.6 | 86.4 | 93.3 | 96.6 | 90.4 | 95.2 | 97.6 | 4.6% | 92.0 | 94.4 | 95.6 | 94.2 | 97.1 | 98.6 | 2.4%
sofa | 69.2 | 76.9 | 81.4 | 70.8 | 79.6 | 84.2 | 70.8 | 79.6 | 84.1 | 72.2 | 81.5 | 86.9 | 2.0% | 52.9 | 54.2 | 54.8 | 80.1 | 89.1 | 94.9 | 51.4%
table | 65.3 | 68.8 | 70.6 | 65.3 | 69.4 | 71.4 | 70.5 | 75.3 | 77.7 | 77.4 | 83.3 | 86.3 | 9.8% | 28.6 | 29.4 | 29.8 | 78.2 | 90.0 | 94.5 | 173.4%
vicon_light | 69.4 | 76.0 | 79.6 | 72.7 | 81.1 | 85.6 | 73.2 | 82.0 |  86.6 | 71.8 | 81.3 | 86.4 | -1.9% | 66.5 | 70.1 | 72.1 | 74.5 | 80.3 | 83.3 | 12.0%
large_loop | 69.2 | 75.1 | 78.1 | 69.9 | 75.8 | 78.8 | 73.2 | 79.3 | 82.4 | 78.6 | 86.2 | 90.3 | 7.4% | 42.5 | 45.1 | 46.4 | 77.7 | 86.8 | 91.5 | 82.8%
Overall | 69.7 | 75.8 | 79.0 | 70.9 | 77.8 | 81.4 | 71.5 | 79.0 | 83.0 | 75.3 | 83.0 | 87.1 | 5.3% | 58.2 | 61.2 | 62.8 | 76.7 | 83.5 | 87.2 | 31.8%
Table 2: Pose Estimation Accuracy on ETH3D-SLAM. The best results are
highlighted in bold. The column “SGP” is the accuracy of the finetuned model
of CAPS using the self-supervised SGP training scheme. The columns “CAPS” and
“DKM” are the accuracies of the pretrained models from their original works.
The columns “iCAPS’ and “iDKM” are accuracies of CAPS and DKM trained by our
iMatching self-supervised learning, with percentage of improvements under 5°
threshold calculated in “%$\uparrow$".

###  4.2 Feature Matching

##### Evaluation Protocol

For each method, we report the mean matching accuracy (MMA), _i.e_., the
percentage of predicted correspondences with reprojection errors less than 1,
2, and 5 pixels. The ground truth correspondences are computed based on the
TartanAir’s pose and depth map.

##### Baselines

We compare against most of the widely used and state-of-the-art models. We
divide them into three categories: (a) non-differentiable models including
SIFT [17], ORB [20], SuperGlue [21], and R2D2 [19], (b) the state-of-the-art
models including CAPS [32], Patch2Pix [41], and ASpanFormer [1], and (c) self-
supervised method, SGP [37]. SGP is specifically designed for CAPS [32]. For
fairness, both SGP and our iMatching are initialized with the exact same
pretrained weights as provided by [32].

##### Results

We report the MMA of selected methods in Table 1. It can be seen that methods
trained with iMatching greatly outperform previous handcrafted, supervised,
and self-supervised methods. Comparing categories (a) and (b), we demonstrate
that self-supervision can further enhance data-driven methods, which already
outperform handcrafted features in our zero-shot scenario except for
Patch2Pix.

Compared with the pretrained supervised methods in (b), iCAPS in (d) offers
1.82$\times$ of the accuracy of the pretrained CAPS at 1-pixel error
tolerance. Figure 5 compares the CAPS and iCAPS model on simple point cloud
registration tasks, with ground truth depth scan used to highlight pose
estimation accuracy. The pretrained model leaves obvious misalignments even if
BA is used to promote consensus of camera poses, which indicates flaws in both
the model accuracy and robustness of a normal BA, but our bilevel optimization
successfully eliminates the artifacts. Patch2Pix and ASpanFormer are two
models showing weak and strong zero-shot performance, respectively. Our iASpan
is capable of improving the already strong ASpanFormer model by 13.6%, and
more qualitative results are available in Figure 6. Additionally, we observe a
strong performance boost of 8.9$\times$ on Patch2Pix, even though it’s a
poorly performed model initially. This shows that our bilevel optimization
formulation is robust against large numbers of outliers.

In addition, our method outperforms SGP by a 5% higher performance gain, as
indicated by (c) and iCAPS in (d). Our method has a relatively obvious
advantage on Natural scenes. They contain repeating texture patterns and
complex lighting conditions, _e.g_. the vegetation in the 4 sample in Figure
6. These conditions typically cause frequent mismatched correspondence and
thus highlight the importance of outlier rejection capability in self-
supervised systems. We conclude that our experiments on CAPS, Patch2Pix, and
ASpanFormer show the strong adaptability of iMatching by covering major types
of end-to-end differentiable models.

Figure 6: iASpan Qualitative Results. Each row shows an uninterrupted video
with 10% of the correspondences. Best viewed digitally. Figure 7: iDKM
Qualitative Results. The top rows (1, 3) show raw image pairs $I_{i}$ and
$I_{j}$. These qualitative results contain restored RGB pixel values of one
image by transferring pixel color given the estimated feature correspondences
from the other image. Our iDKM accurately identifies matching on the features
best explained by object shape. Best viewed digitally.

###  4.3 Relative Pose Estimation

We next conduct another task, relative pose estimation, which is a crucial
application in computer vision, to demonstrate the adaptability of the
iMatching training scheme. We follow [1] to recover the camera pose by solving
the essential matrix from the predicted correspondences using RANSAC.

##### Baselines

To evaluate the performance gain after training, we use both expectation- and
regression-based methods with pre-trained weights, as represented by CAPS [32]
and DKM [5], respectively. Again, we compare against SGP [37] by training CAPS
with iMatching. The non-differentiable SuperGlue model [21] is included for
reference.

##### Accuracy

A pose estimate is considered to be correct if the angular error in both
rotation and translation falls below thresholds of 5∘, 10∘, and 20∘, and we
report the area-under-curve (AUC) of the pose error in Table 2. iMatching
shows superior performance over supervised and self-supervised methods. Data-
driven methods perform strongly, except for the DKM. Despite demonstrating the
strongest performance under supervised setting on MegaDepth and ScanNet [5],
DKM is behind SuperGlue and CAPS by a large margin on the unseen ETH3D-SLAM
scenes. However, iMatching is able to boost its accuracy by 30% compared with
the pretrained, making it the strongest among all methods. From image warping
results in Figure 7, it also demonstrates satisfying per-pixel matching
quality.

##### Comparison with SGP

Pose estimation accuracy improvement brought by SGP is only observed in 5
scenes and is overall negligible. This is primarily because the CAPS model has
strong generalizability in the task but the simple supervision formulation in
CAPS already has saturated performance under the evaluation metrics: SGP
training can only encourage overall consensus between correspondence following
the epipolar constraint regardless of actual 3D shape, so the effectiveness of
SGP is not as obvious as what we observed in feature matching. In contrast,
our design allows us to further improve the already strong CAPS model by 5%.
Our iDKM experiment also verifies that iMatching is less likely to be
vulnerable to initially worse pose estimation accuracy, as its iterative BA
process brings additional robustness to correct error that occurs in map
initialization. Therefore, we conclude that our imperative learning works
pragmatically and effectively, thanks to its carefully designed robust BA.

##  5 Conclusion

We explore an exciting new direction in self-supervised feature correspondence
learning, _i.e_. imperative learning, using bundle adjustment as a supervision
signal, formulated as a bilevel optimization problem, and learning feature
correspondence from uninterrupted video sequence without any form of ground
truth labels. We highlight the simple and efficient optimization technique
used to differentiate through the complex bundle adjustment process. We
experimented with the state-of-the-art models and demonstrated obvious
performance leaps in tasks including feature matching and relative pose
estimation, empirically verifying the importance of our robust bundle
adjustment. This paper makes an attractive starting point for integrating the
learning process of feature correspondence to downstream applications such as
visual odometry and 3D reconstruction in an online manner.

## Acknowledgements

This work was in part supported by the ONR award N00014-24-1-2003. Any
opinions, findings, conclusions, or recommendations expressed in this paper
are those of the authors and do not necessarily reflect the views of the ONR.
The authors wish to express their gratitude for the generous gift funding
provided by Cisco Systems Inc. and InnoPeak Technology Inc.

## References

  * Chen et al. [2022] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David McKinnon, Yanghai Tsin, and Long Quan.  Aspanformer: Detector-free image matching with adaptive span transformer.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , pages 20–36. Springer, 2022.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * DeTone et al. [2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 224–236, 2018.
  * Dusmanu et al. [2019] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-net: A trainable cnn for joint detection and description of local features.  _arXiv preprint arXiv:1905.03561_ , 2019.
  * Edstedt et al. [2023] Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg.  DKM: Dense kernelized feature matching for geometry estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Fu et al. [2023] Taimeng Fu, Shaoshu Su, and Chen Wang.  iSLAM: Imperative SLAM.  _arXiv preprint arXiv:2306.07894_ , 2023.
  * Hartley and Zisserman [2003] Richard Hartley and Andrew Zisserman.  _Multiple view geometry in computer vision_.  Cambridge university press, 2003.
  * Jatavallabhula et al. [2020] Krishna Murthy Jatavallabhula, Ganesh Iyer, and Liam Paull.  $\triangledown$slam: Dense slam meets automatic differentiation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2130–2137. IEEE, 2020.
  * Ji et al. [2021] Kaiyi Ji, Junjie Yang, and Yingbin Liang.  Bilevel optimization: Convergence analysis and enhanced design.  In _International conference on machine learning_ , pages 4882–4892. PMLR, 2021.
  * Kasper et al. [2019] Mike Kasper, Steve McGuire, and Christoffer Heckman.  A Benchmark for Visual-Inertial Odometry Systems Employing Onboard Illumination.  In _Intelligent Robots and Systems (IROS)_ , 2019.
  * Keetha et al. [2022] Nikhil Varma Keetha, Chen Wang, Yuheng Qiu, Kuan Xu, and Sebastian Scherer.  AirObject: A temporally evolving graph embedding for object identification.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Lepetit et al. [2009] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua.  Ep n p: An accurate o (n) solution to the p n p problem.  _International journal of computer vision_ , 81:155–166, 2009.
  * Li et al. [2020a] Shunkai Li, Xin Wang, Yingdian Cao, Fei Xue, Zike Yan, and Hongbin Zha.  Self-supervised deep visual odometry with online adaptation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6339–6348, 2020a.
  * Li et al. [2020b] Xinghui Li, Kai Han, Shuda Li, and Victor Prisacariu.  Dual-resolution correspondence networks.  _Advances in Neural Information Processing Systems_ , 33:17346–17357, 2020b.
  * Li and Snavely [2018] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2041–2050, 2018.
  * Lopes et al. [2022] Alexandre Lopes, Roberto Souza, and Helio Pedrini.  A survey on rgb-d datasets.  _arXiv preprint arXiv:2201.05761_ , 2022.
  * Lowe [2004] David G. Lowe.  Distinctive image features from scale-invariant keypoints.  _Int. J. Comput. Vision_ , 60(2):91–110, 2004.
  * Luo et al. [2018] Zixin Luo, Tianwei Shen, Lei Zhou, Siyu Zhu, Runze Zhang, Yao Yao, Tian Fang, and Long Quan.  Geodesc: Learning local descriptors by integrating geometry constraints.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 168–183, 2018.
  * Revaud et al. [2019] Jerome Revaud, Philippe Weinzaepfel, César De Souza, Noe Pion, Gabriela Csurka, Yohann Cabon, and Martin Humenberger.  R2d2: repeatable and reliable detector and descriptor.  _arXiv preprint arXiv:1906.06195_ , 2019.
  * Rublee et al. [2011] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R. Bradski.  Orb: An efficient alternative to sift or surf.  _2011 International Conference on Computer Vision_ , pages 2564–2571, 2011.
  * Sarlin et al. [2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4938–4947, 2020.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. [2016] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Schonberger et al. [2017] Johannes L Schonberger, Hans Hardmeier, Torsten Sattler, and Marc Pollefeys.  Comparative evaluation of hand-crafted and learned local features.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1482–1491, 2017.
  * Schöps et al. [2019] Thomas Schöps, Torsten Sattler, and Marc Pollefeys.  BAD SLAM: Bundle adjusted direct RGB-D SLAM.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Shi et al. [2019] Yunxiao Shi, Jing Zhu, Yi Fang, Kuochin Lien, and Junli Gu.  Self-supervised learning of depth and ego-motion with differentiable bundle adjustment.  _arXiv preprint arXiv:1909.13163_ , 2019.
  * Stathopoulou et al. [2019] Elisavet Konstantina Stathopoulou, M Welponer, and Fabio Remondino.  Open-source image-based 3d reconstruction pipelines: Review, comparison and evaluation.  _The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XLII-2/W17_ , pages 331–338, 2019.
  * Sun et al. [2021] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  Loftr: Detector-free local feature matching with transformers.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8922–8931, 2021.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _European conference on computer vision_ , pages 402–419. Springer, 2020.
  * Teed and Deng [2021] Zachary Teed and Jia Deng.  Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras.  _Advances in Neural Information Processing Systems_ , 34:16558–16569, 2021.
  * Wang et al. [2022] Chen Wang, Yuheng Qiu, Dasong Gao, and Sebastian Scherer.  Lifelong graph learning.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Wang et al. [2020a] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely.  Learning feature descriptors using camera pose supervision.  In _European Conference on Computer Vision_ , pages 757–774. Springer, 2020a.
  * Wang et al. [2020b] Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu Hu, Ashish Kapoor, and Sebastian Scherer.  Tartanair: A dataset to push the limits of visual slam.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 4909–4916, 2020b.
  * Wu [2013] Changchang Wu.  Towards linear-time incremental structure from motion.  In _2013 International Conference on 3D Vision-3DV 2013_ , pages 127–134. IEEE, 2013.
  * Xu et al. [2023] Kuan Xu, Yuefan Hao, Shenghai Yuan, Chen Wang, and Lihua Xie.  AirVO: An illumination-robust point-line visual odometry.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2023.
  * Yang et al. [2023] Fan Yang, Chen Wang, Cesar Cadena, and Marco Hutter.  iplanner: Imperative path planning.  In _Robotics: Science and Systems (RSS)_ , 2023.
  * Yang et al. [2021] Heng Yang, Wei Dong, Luca Carlone, and Vladlen Koltun.  Self-supervised geometric perception.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14350–14361, 2021.
  * Yi et al. [2016] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In _European conference on computer vision_ , pages 467–483. Springer, 2016.
  * Zhan et al. [2023] Zitong Zhan, Xiangfu Li, Qihang Li, Haonan He, Abhinav Pandey, Haitao Xiao, Yangmengfei Xu, Xiangyu Chen, Kuan Xu, Kun Cao, Zhipeng Zhao, Zihan Wang, Huan Xu, Zihang Fang, Yutian Chen, Wentao Wang, Xu Fang, Yi Du, Tianhao Wu, Xiao Lin, Yuheng Qiu, Fan Yang, Jingnan Shi, Shaoshu Su, Yiren Lu, Taimeng Fu, Karthik Dantu, Jiajun Wu, Lihua Xie, Marco Hutter, Luca Carlone, Sebastian Scherer, Daning Huang, Yaoyu Hu, Junyi Geng, and Chen Wang.  PyPose v0.6: The imperative programming interface for robotics.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) Workshop_ , 2023.
  * Zhao et al. [2023] Shibo Zhao, Damanpreet Singh, Haoxiang Sun, Rushan Jiang, YuanJun Gao, Tianhao Wu, Jay Karhade, Chuck Whittaker, Ian Higgins, Jiahe Xu, Yuheng Qiu, Sourojit Saha, Chen Wang, Wenshan Wang, and Sebastian Scherer.  Subt-mrs: A subterranean, multi-robot, multi-spectral and multi-degraded dataset for robust slam.  _arXiv preprint arXiv:2307.07607_ , 2023.
  * Zhou et al. [2021] Qunjie Zhou, Torsten Sattler, and Laura Leal-Taixe.  Patch2pix: Epipolar-guided pixel-level correspondences.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4669–4678, 2021.