# DeDoDe: Detect, Don’t Describe — Describe, Don’t Detect
for Local Feature Matching

Johan Edstedt1 Georg Bökman2 Mårten Wadenbäck1 Michael Felsberg1
1Linköping University, 2Chalmers University of Technology

###### Abstract

Keypoint detection is a pivotal step in 3D reconstruction, whereby sets of (up
to) K points are detected in each view of a scene. Crucially, the detected
points need to be consistent between views, i.e., correspond to the same 3D
point in the scene. One of the main challenges with keypoint detection is the
formulation of the learning objective. Previous learning-based methods
typically jointly learn descriptors with keypoints, and treat the keypoint
detection as a binary classification task on mutual nearest neighbours.
However, basing keypoint detection on descriptor nearest neighbours is a proxy
task, which is not guaranteed to produce 3D-consistent keypoints. Furthermore,
this ties the keypoints to a specific descriptor, complicating downstream
usage. In this work, we instead learn keypoints directly from 3D consistency.
To this end, we train the detector to detect tracks from large-scale SfM. As
these points are often overly sparse, we derive a semi-supervised two-view
detection objective to expand this set to a desired number of detections. To
train a descriptor, we maximize the mutual nearest neighbour objective over
the keypoints with a separate network. Results show that our approach, DeDoDe,
achieves significant gains on multiple geometry benchmarks. Code is provided
at github.com/Parskatt/DeDoDe.

##  1 Introduction

Incremental Structure from Motion (SfM) can roughly be divided into detection,
matching, and estimation. In the detection stage, we seek to detect keypoints
in the 3D scene that are consistent between views. This is a crucial step, as
the optimization process relies on accurate 3D tracks, _i.e_., 3D points
consistently detected in multiple views. While keypoints are necessary for
SfM, it is difficult to formulate a learning-based objective. Traditional
methods often rely on proxy tasks such as extrema in differences of Gaussians
[19], corner detection [12], while learning-based methods often aim to detect
mutual nearest neighbours of descriptors [32, 10]. While these methods can
result in distinct and reproducible points, they do not directly optimize 3D
consistency of the points between views.

Figure 1: DeDoDe closes the matching gap. There is a large gap in performance
between traditional detector/descriptor models and fully-fledged matching
models. DeDoDe significantly reduces this gap, demonstrating the untapped
potential of the simple detector+descriptor approach.

In order to tackle this discrepancy, we propose to directly learn 3D
consistency by detecting 3D tracks. To do this we use SfM reconstructions
directly as supervision.

Figure 2: Qualitative comparison. Left: Matches from the joint
detector/descriptor DISK [32], Right: Matches from our decoupled
detector/descriptor DeDoDe. DeDoDe shows substantial improvements in both
detector precision and descriptor quality in comparison to previous
approaches, performing well even under significant changes in viewpoint.

Since we have access to ground truth 3D tracks, we can directly optimize to
detect these. However, as the 3D tracks are produced by a base detector, there
is a risk of missing reliable keypoints discarded by the original detector. To
mitigate this problem, we propose a semi-supervised two-view top-$k$
consistency objective, which enables us to surpass the initial detector both
in precision and recall. A detailed description of our approach is provided in
Section 4.

When training detectors and descriptors jointly, the descriptor objective is
complex. In contrast, we can directly optimize the mutual nearest neighbour
negative log-likelihood for the descriptions over DeDoDe keypoints to train
the DeDoDe descriptor, which we will refer to as DeDoDe-B. While this model
performs well, local features struggle with describing repeating structures,
which requires a larger context. Recent work [9] has shown promising results
of using globally informed DINOv2 [22] features for dense matching. We show
that incorporating frozen DINOv2 features into a larger version of our
descriptor, which we call DeDoDe-G, also increases performance. Our approach
to descriptor learning is detailed in Section 5.

Finally, we perform an extensive set of SotA experiments and ablations in
Section 6, in which we find that DeDoDe sets a new state-of-the-art in feature
matching. Our approach bridges the gap between the traditional
detector+descriptor approach to matching and modern end-to-end matchers as can
be seen in Figure 1. A qualitative comparison between the joint
detector/descriptor DISK [32] and DeDoDe is presented in Figure 2.

Figure 3: Previous detector/descriptor approaches compared to DeDoDe. Left:
Traditional methods are typically two-stage, first detecting promising
keypoints with a detector $f_{\theta}$, which are subsequently described by a
descriptor $g_{\theta}$. This approach suffers from the lack of synergy
between the detector and descriptor [30, 7]. Right: Later, learning-based
methods using the joint learning paradigm were introduced. Here the objectives
of the descriptor and the detector are aligned. However, this has other
issues. One issue is that introducing inter-dependencies of the
descriptor/detector often degrades performance [16]. For example, repeating
complex patterns may constitute excellent keypoints, but fail to match, while
unique textureless regions may match, but do not produce exact localization.
Middle: In this paper we take a new approach and decouple the descriptor and
detector learning while having aligned objectives. We do this by learning to
detect and match 3D tracks from large-scale SfM. In Section 4 we describe how
we create a training objective for the detector, and in Section 5 we describe
the approach for the descriptor. Our approach leads to significant
improvements to the state-of-the-art, as demonstrated in Section 6, showing
the potential of detector based methods.

##  2 Related Work

Keypoint Detection from Descriptors: Dusmanu et al. [7] proposed a joint
descriptor/detector pipeline. Revaud et al. [25] extended these ideas and
proposed a set of auxiliary losses to improve the detector. Tyszkiewicz et al.
[32] proposed DISK, which uses a joint training objective for keypoint
detection and description through policy gradient optimization, enabling
backpropagation to the detector. Zhao et al. [38, 39] proposed ALIKE and
ALIKED, where a local softmax weighting approach is used for keypoint
detection that enables backpropagation to the detector.

Keypoint Detection from a Base Detector: Closest to our work are keypoint
detectors that aim to surpass a base detector. One of the earliest works TaSK
[28] classified keypoints that are consistent over long time periods, an idea
that was further improved in TILDE [33]. A similar approach was investigated
by Hartmann et al. [13], where a classifier was trained to discriminate SfM
matchable keypoints. LIFT [36] crops small patches around SfM tracks, but does
not optimize for detecting tracks directly. SuperPoint [6] is trained from an
initial base detector by means of a synthetic training set of corners. A
second detector is then trained to produce consistent detections of the base
detector for heavily augmented images. More recent approaches, such as SOLD2
[23] and DeepLSD [24] further develop these ideas for line detection. Our work
can be seen as further development of these ideas with some key differences.
In contrast to previous work we introduce a top-$k$ two-view semi-supervised
detection objective combined with a coverage objective. Furthermore, we use an
aligned but decoupled detector/descriptor learning approach. See Section 4.

Whether to Couple or Decouple Keypoint Detection and Description:
Traditionally, handcrafted feature detection pipelines, _e.g_. SIFT [19], the
keypoint detection is independent of the descriptors. Early learned work in a
similar fashion followed the decoupled approach [20, 21, 2]. However, this
approach has been called into question in later works [27, 30, 7], with the
reasoning that descriptors often succeed in matching where keypoints fail.
Hence most following work has worked in the joint learning setting [25, 32,
10], which binds the keypoints to a specific descriptor. Recently, Li et al.
[16], argue that the joint training is detrimental, as poor descriptors lead
to unreliable mutual nearest neighbours, resulting in worse keypoint
detectors. However, they still use the descriptor features as input to the
detector network. In contrast to joint previous works, we consider an aligned
objective that is still decoupled.

##  3 Why Decouple Keypoint Detection and Description?

In this paper we argue for a descriptor-agnostic approach to keypoint
detection, see Figure 3. There are multiple reasons for this, we summarize the
main points below

  1. 1.

Compatability: Mutual nearest neighbours of a certain descriptor does not
guarantee general matchability or consistency. By being agnostic to the
descriptor, our keypoints can be used for arbitrary matchers, as we
demonstrate in Table 1.

  2. 2.

Decoupling Gains: Thinking in a modular fashion lets us design decoupled
experiments, for example Table 2 and 3, as well as Table 6.

  3. 3.

Performance: Our approach improves significantly on the state-of-the-art, see
Figure 1. This demonstrates the soundness of our approach.

The following sections describe our method. In Section 4 we introduce the
DeDoDe detector, and in Section 5 the DeDoDe descriptor.

##  4 Detect, Don’t Describe

For an overview of our approach, see Figure 4.

###  4.1 Preliminaries

Let $f_{\theta}(x|I)$ be a mapping producing a log density (possibly non-
normalized) over an image $I$. We would like this network to produce high
values for all 3D consistent keypoints, and low values otherwise. Now let

$\mathcal{D}=\\{I^{j}\\}_{j=1}^{J}$ (1)

be a dataset of images with cardinality $|\mathcal{D}|=J$. We will denote the
set of keypoints for a given image $I^{j}$ in pixel coordinates as
$\textbf{x}^{j}_{\rm kp}=\\{x^{j}_{i}\\}_{i=1}^{K^{j}}$ where $K^{j}$ is the
number of observable keypoints in image $I^{j}$. Formally, we want to maximize
the following log-likelihood objective,

$\max_{\theta}\sum_{j=1}^{|\mathcal{D}|}\sum_{i=1}^{K^{j}}f_{\theta}(x_{i}^{j}|I^{j})-\log
Z_{\theta}(I^{j}),$ (2)

where we have defined

$Z_{\theta}(I^{j}):=\sum_{x^{j}\in I^{j}}\exp(f_{\theta}(x^{j}|I^{j})).$ (3)

That is, we want to maximize the likelihood of sampling 3D keypoints. For
notational convenience, we will also define

$p_{f_{\theta}}(x_{i}^{j}):=\exp(f_{\theta}(x_{i}^{j}|I^{j})-\log
Z_{\theta}(I^{j}))$ (4)

The main challenge of keypoint detection is how to maximize Equation 2. In
particular, in contrast to other keypoint tasks, there are no ground truth
annotations. In this paper we will consider a good prior for a keypoint as a
2D detection (from a base keypoint detector) that has “survived” 3D
reconstruction, _i.e_., formed a 3D track.

Figure 4: Training objective of DeDoDe for keypoint detection. We sample an
image pair from a large-scale SfM + MVS reconstruction. (i) For this pair we
find all covisible detections (from a base detector) that resulted in a 3D
track. We take the union of tracks detected in the image (indicated by filled
lines in the figure), and tracks not detected but covisible (indicated by
dotted lines in the figure). See Section 4.2. (ii) We project these detections
into the images. We then convert this point set into a softened distribution,
which constitutes our prior for good keypoints, see Section 4.3. (iii) We then
condition this distribution with the predictions of DeDoDe, as the base
detector may have false negatives, see Section 4.4. (iv) Finally, we threshold
this distribution at the $k$:th highest value, and compute the cross-entropy
between DeDoDe and the target. Using top-k stabilizes the self-supervised
objective. See Section 4.5.

###  4.2 Keypoints as 3D Tracks

We use a dataset of large-scale SfM reconstructions, MegaDepth [17], where
SIFT [19] is used as keypoint detector, _i.e_., scale space extrema. These
detections go through a filtering process, whereby only 3D consistent
detections remains after optimization finishes. However, in each image, only a
fraction of all visible tracks are detected. To solve this issue we propose to
sample pairs of images $I^{\mathcal{A}}$ and $I^{\mathcal{B}}$, and use the
union of covisible detections. We obtain the covisibility by means of Multi-
View Stereo depth maps.

###  4.3 A Smooth Keypoint Detection Prior

Due to the detector producing detections on a discrete grid, and for reasons
we will discuss in Section 4.4, a smooth detection prior is desirable. We do
this by a 2-stage process. For both $I^{\mathcal{A}}$ and $I^{\mathcal{B}}$ we
start by placing Dirac deltas $\delta(\mathbf{x}_{\rm kp})$ at the nearest
grid position of the detections. We then convolve these with a Gaussian kernel
with $\sigma=0.5$ pixels, and add a small constant that corresponds to a
uniform distribution. Mathematically,

$\displaystyle\tilde{p}_{\rm
kp}^{\mathcal{A}}\propto\mathcal{N}(0,\sigma^{2})*\delta(\mathbf{x}_{\rm
kp}^{\mathcal{A}})+C^{\mathcal{A}},$ (5) $\displaystyle\tilde{p}_{\rm
kp}^{\mathcal{B}}\propto\mathcal{N}(0,\sigma^{2})*\delta(\mathbf{x}_{\rm
kp}^{\mathcal{B}})+C^{\mathcal{B}}.$ (6)

Finally we warp the distributions of $\mathcal{A},\mathcal{B}$ using the depth
to produce $\tilde{p}_{\rm kp}^{\mathcal{A}\to\mathcal{B}},\tilde{p}_{\rm
kp}^{\mathcal{B}\to\mathcal{A}}$. Combining yields

$\displaystyle p^{\mathcal{A}}_{\rm kp}\propto\tilde{p}_{\rm
kp}^{\mathcal{A}}\cdot\tilde{p}_{\rm kp}^{\mathcal{B}\to\mathcal{A}},$ (7)
$\displaystyle p^{\mathcal{B}}_{\rm kp}\propto\tilde{p}_{\rm
kp}^{\mathcal{B}}\cdot\tilde{p}_{\rm kp}^{\mathcal{A}\to\mathcal{B}}.$ (8)

In practice, the constants in equations 5 and 6 are implicitly defined by
approximating $\log p_{\rm kp}\approx\vartheta p_{\rm kp}$, with
$\vartheta=50$. The end result is a detection prior which produces
particularly high values at 3D-tracks detected in _both_ images, which we
found beneficial.

Note that while this is a simple and powerful way of producing a detection
prior, there is a multitude of ways one could construct it. For example, we
could exclude all tracks smaller than a certain track length, or only include
tracks with sufficiently small reprojection errors. However, such approaches
introduce more complex hyperparameters, and we found our simple approach to
already yield state-of-the-art results.

###  4.4 Posterior Detection Distribution

While the detection prior yields good detections, we found that it is often
overly sparse in practice. This is due to the base detector not detecting all
keypoints, _i.e_., insufficient recall. To amend this issue, we propose to
update the prior distribution with the predictions of $f_{\theta}$, _i.e_.,

$p\propto p_{f_{\theta}}\cdot p_{\rm kp}.$ (9)

This enables the network to detect keypoints missed by the base detector. Note
that while it would appear that multiplying the distributions cannot add
detections, the fact that we have put a small flat prior on the keypoint
distribution in practice creates peaks around the model detections that enable
detection of additional keypoints.

###  4.5 Detector Training Objective

In principle, one could form an objective directly using the cross entropy
between $p_{f_{\theta}}$ and $p$. However, we found training to be more stable
when thresholding and binarizing $p$ in practice. This is due to the non-
binary objective having degenerate solutions, as we show in Suppl. F. To this
end, we select the top-$k$ detections in a batch, where we empirically select
$k={\rm batchsize}\cdot 1024$, to construct the target distributions. Then we
simply compute the cross entropy between the predicted distribution and the
target as a loss,

$\mathcal{L}_{\rm detection}={\rm CE}(p_{f_{\theta}},p_{\text{top-}k}).$ (10)

We additionally add a coverage regularization on $f_{\theta}$ by computing the
cross-entropy between a low-pass filtered $p_{f_{\theta}}$ and a binary
distribution $p_{\rm MVS}$ indicating successful MVS reconstruction, _i.e_.,

$\mathcal{L}_{\rm coverage}={\rm
CE}(\mathcal{N}(0,\sigma^{2})*p_{f_{\theta}},\mathcal{N}(0,\sigma^{2})*p_{\rm
MVS}).$ (11)

In practice we used $\sigma=12.5$ pixels. This regularization is necessary, as
the network may otherwise produce detections in unmatchable regions (_e.g_.
sky). Our combined loss is

$\mathcal{L}=\mathcal{L}_{\rm detection}+\mathcal{L}_{\rm coverage}.$ (12)

As with our choice of prior distribution, these choices are not guaranteed to
be optimal, but worked well in practice. Note that the choice of $k$ depends
on the expected overlap between the views. For difficult pairs, one would
expect fewer keypoints to overlap. It is in principle possible to control the
sparsity of the detector by decreasing or increasing $k$, but investigation of
this is outside the scope of the paper.

###  4.6 Sampling Keypoints at Inference

To sample keypoints, we follow previous work and select the top $K$ most
likely keypoints from $f_{\theta}$. We follow SiLK [10] and do not employ non-
max supression.

###  4.7 Network Architecture

Encoder: We use a VGG-19 network pretrained on ImageNet, implemented in
torchvision. We use strides $[1,2,4,8]$. The output at each scale is a
feature-map with $[64,128,256,512]$ channels respectively.

Decoder: We use the architecture of the depthwise convolution refiners
proposed in DKM [8], with 8 blocks per scale and internal dimension
$[64,128,256,512]$. The input at each scale is the encoded features, as well
as context from previous decoding layers. The output at each scale is a
residual addition to a dense grid of keypoint logits. Between each scale these
logits are upsampled using bicubic interpolation.

Note that our network is different from DKM which takes two images as input,
constructing a local correlation volume + stacked feature maps, while ours
simply refines the features.

##  5 Describe, Don’t Detect

###  5.1 Descriptor Preliminaries

Given two sets of keypoints, $\textbf{x}^{\mathcal{A}}_{\rm kp}$,
$\textbf{x}^{\mathcal{B}}_{\rm kp}$, and the respective images
$I^{\mathcal{A}},I^{\mathcal{B}}$, the task of a descriptor network
$\mathbf{g}_{\theta}(x|I^{\mathcal{A}})$ is to produce descriptions
$\mathbf{g}_{\theta}(\textbf{x}^{\mathcal{A}}_{\rm
kp}|I^{\mathcal{A}}),\mathbf{g}_{\theta}(\textbf{x}^{\mathcal{B}}_{\rm
kp}|I^{\mathcal{B}})$ that match well for keypoints corresponding to the same
3D point, and match poorly for non-corresponding keypoints.

We choose the probabilistic framework, as is common [32, 10], where

$p_{g_{\theta}}(x^{\mathcal{A}}|x^{\mathcal{B}}):=\frac{\exp(\langle\mathbf{g}_{\theta}(x^{\mathcal{A}}|I^{\mathcal{A}}),\mathbf{g}_{\theta}(x^{\mathcal{B}}|I^{\mathcal{B}})\rangle)}{\sum_{x_{i}^{\mathcal{A}}\in\textbf{x}^{\mathcal{A}}_{\rm
kp}}\exp(\langle\mathbf{g}_{\theta}(x_{i}^{\mathcal{A}}|I^{\mathcal{A}}),\mathbf{g}_{\theta}(x^{\mathcal{B}}|I^{\mathcal{B}})\rangle)},$
(13)

defines the conditional matching distribution, and formally define matching
likelihoods as

$L_{g_{\theta}}(x^{\mathcal{A}},x^{\mathcal{B}}):=p_{g_{\theta}}(x^{\mathcal{B}}|x^{\mathcal{B}})\cdot
p_{g_{\theta}}(x^{\mathcal{B}}|x^{\mathcal{A}}).$ (14)

This objective has the benefit of working well with mutual nearest neighbours
based downstream matchers.

###  5.2 Descriptor Training Objective

Taking the logarithm of equation (14) leads to a straightforward log-
likelihood objective

$\ell_{g_{\theta}}(x^{\mathcal{A}},x^{\mathcal{B}})=\log
p_{g_{\theta}}(x^{\mathcal{A}}|x^{\mathcal{B}})+\log
p_{g_{\theta}}(x^{\mathcal{B}}|x^{\mathcal{A}}).$ (15)

The main difficulty with optimizing this objective is the normalizing term,
which in most works is either approximated [32], or brute-force optimized in
lower resolution [10]. However, as our formulation is decoupled, we face no
such issues and simply use trained DeDoDe keypoints. During training,
similarly to inference, we sample $K=5000$ descriptions from a dense grid,
using a pretrained DeDoDe detector.

###  5.3 Descriptor Network Architecture

The architecture of the descriptor largely matches that of the detector.
Importantly, however, they do not share any weights.

Encoder: We use a VGG-19 network pretrained on ImageNet, implemented in
torchvision. We use features at strides $[1,2,4,8]$. The output at each scale
is a feature-map with $[64,128,256,512]$ channels respectively.

Decoder: We use the architecture of the depthwise convolution refiners
proposed in DKM [8], with 5 blocks per scale and internal dimension
$[32,64,256,512]$. We call this model DeDoDe-B. The input at each stride is
the encoded features, as well as context from previous decoding layers. The
output at each scale is a residual addition to a dense grid of descriptions.
Between each scale the descriptions are upsampled using bilinear
interpolation. We use a description dimension of 256. We normalize the
descriptions, and use a temperature of $20^{-1}$ during both training and
inference.

Note, as for the detector, that our network is different from DKM which takes
two images as input, constructing a local correlation volume + stacked feature
maps, while ours simply refines the features of one image.

DeDoDe-G: To explore the full potential of detector descriptor methods, we
propose to incorporate features from DINOv2 [22]. This addition is simple in
our framework, we just add frozen DINOv2 features at stride 14, with a
corresponding decoder stage. We use an internal dimension of $768$ for this
decoder. With this simple addition, we achieve impressive performance gains as
demonstrated in Table 1 and Table 4.

##  6 Experiments

###  6.1 Training

DeDoDe Detector: We train for $100000$ steps of batch size 8 on the MegaDepth
dataset, using the same training split as DKM [8]. We use a fixed image size
of $512\times 512$. Training is done on a single A100 GPU, and takes about 30
hours.

DeDoDe Descriptor: Mirroring our detector training setup, we train for
$100000$ steps of batch size 8 on the MegaDepth dataset. We use a single A100
GPU, and training takes about 24 hours. The slightly shorter training time
comes from using fewer blocks in the decoder.

Figure 5: Qualitative example of DeDoDe matches. We picked a random pair in
the MegaDepth-1500 testset and visualize confident DeDoDe matches. As can be
seen in the figure, DeDoDe produces reliable keypoints and good matches, even
in challenging situations. There are still outliers however, particularly for
repeating structures such as similar statues, and identical pillars. Best
viewed in high resolution.

###  6.2 Inference

For all SotA experiments we run at a resolution of $784\times 784$ and sample
top-$k$ keypoints.

###  6.3 Integrating DeDoDe With Matchers

Recent works [29, 8] often claim that detection localization errors hinder
performance. To investigate this claim, we integrate DeDoDe into the recent
SotA matcher RoMa [9]. We integrate RoMa into our detection framework by
sampling the dense warp of RoMa at the location of DeDoDe keypoints, and match
each keypoint by warping it to the other image and finding its nearest-
neighbour DeDoDe keypoint there.

As expected, quantizing the RoMa warp to DeDoDe keypoints decreases
performance compared to the results of RoMa, but increases the results
compared to our detector+descriptor approach. If we view RoMa as an oracle, we
can view the detector gap as about 7 points AUC@$5^{\circ}$. This indicates
that there are still significant gains that can be made by improving the
detector further.

###  6.4 Qualitative Examples

DeDoDe Keypoints: Here we qualitatively demonstrate the types of keypoints and
matches learned by DeDoDe. In Figure 5 we show DeDoDe matches on a non-
cherrypicked, _i.e_. randomly chosen, pair from the MegaDepth-1500 testset. We
find that DeDoDe produces repeatable keypoints that match well.

###  6.5 SotA Comparison

MegaDepth-1500 Relative Pose: MegaDepth-1500 is a relative pose benchmark
proposed in LoFTR [29], and consists of 1500 pairs of images in two scenes of
the MegaDepth dataset, which are non-overlapping with our training set. We
compare DeDoDe against previous detector/descriptor models, as well as fully
fledged matchers. We present results in Table 1. Our results clearly show the
soundness of our approach, with gains of +17.8 AUC$@5^{\circ}$ compared to
DISK, +12.9 AUC$@5^{\circ}$ compared to SiLK, and +10.9 AUC$@5^{\circ}$
compared to ALIKED.

Table 1: SotA comparison on the Megadepth-1500 benchmark. The top portion contains dense and detector free matching methods, the middle part keypoint matching methods, while the bottom portion contains traditional MNN matching from keypoints and descriptors. Measured in AUC (higher is better). Method $\downarrow$ AUC $\rightarrow$ | $@5^{\circ}$ | $@10^{\circ}$ | $@20^{\circ}$
---|---|---|---
LoFTR [29] CVPR’21 | 52.8 | 69.2 | 81.2
SE2-LoFTR-4 [3] CVPRW’22 | 52.6 | 69.2 | 81.4
QuadTree [31] ICLR’22 | 54.6 | 70.5 | 82.2
ASpanFormer [5] ECCV’22 | 55.3 | 71.5 | 83.1
ASTR [37] CVPR’23 | 58.4 | 73.1 | 83.8
DKM [8] CVPR’23 | 60.4 | 74.9 | 85.1
PMatch [40] CVPR’23 | 61.4 | 75.7 | 85.7
RoMa [9] Arxiv’23 | 62.6 | 76.7 | 86.3
DeDoDe \+ RoMa | 55.1 | 71.6 | 83.5
SuperGlue [26] CVPR’19 | 49.7 | 67.1 | 80.0
SGMNet [4] CVPR’21 | 43.2 | 61.6 | 75.6
LightGlue [18] ICCV’23 | 49.9 | 67.0 | 80.1
SuperPoint [6] CVPRW’18 | 31.7 | 46.8 | 60.1
DISK [32] NeurIps’20 | 35.0 | 51.4 | 64.9
ALIKED [39] TIM’23 | 41.9 | 58.4 | 71.7
SiLK [10] ICCV’23 | 39.9 | 55.1 | 66.9
DeDoDe-B | 49.4 | 65.5 | 77.7
DeDoDe-G | 52.8 | 69.7 | 82.0

MegaDepth Detector Repeatability:

Table 2: SotA comparison on MegaDepth-Repeatability-10k. *:Non-max supression removed. Note that DeDoDe and SiLK do not use non-max supression. Method | Repeatability @ 10k
---|---
| $0.1\%$ | $0.2\%$ | $0.5\%$
DISK [32] Neurips’20 | 18.4 | 56.6 | 93.0
DISK* [32] Neurips’20 | 32.6 | 60.0 | 83.9
ALIKED [39] IEEE-TIM’23 | 19.4 | 63.3 | 90.8
ALIKED* [39] IEEE-TIM’23 | 26.4 | 63.2 | 85.6
SiLK [10] ICCV’23 | 21.2 | 46.5 | 67.1
DeDoDe | 40.1 | 63.7 | 80.3
Table 3: SotA comparison on MegaDepth-Repeatability-2k. *: Non-max supression removed. Note that DeDoDe and SiLK do not use non-max supression. Method | Repeatability @ 2k
---|---
|  @$0.1\%$ |  @$0.2\%$ |  @$0.5\%$
DISK [32] Neurips’20 | 17.0 | 41.1 | 65.7
DISK* [32] Neurips’20 | 21.8 | 40.1 | 57.5
SiLK [10] ICCV’23 | 8.2 | 21.8 | 35.8
ALIKED [39] IEEE-TIM’23 | 19.3 | 45.4 | 67.9
ALIKED* [39] IEEE-TIM’23 | 21.2 | 46.1 | 66.5
DeDoDe | 24.3 | 41.8 | 58.3

Previous benchmarks for keypoint detectors often jointly measure the
repeatability of keypoints and the matchability of the descriptors into single
metrics. Here we aim to disentangle these two measurements, and _only_ measure
keypoint repeatability. To do this, we use the ground truth warp, and measure
the percentage of inliers, _i.e_., pairs of keypoints close enough, under a
set of thresholds. We investigate two different settings, the many keypoints
setting (maximum 10000 keypoints), and the few keypoint setting (maximum 2000
keypoints). Results are presented in Table 2 and Table 3. The results reveal
interesting properties of detectors. In particular, we find that DeDoDe and
SiLK show large gains from higher numbers of keypoints for tight thresholds,
while DISK shows large gains in loose thresholds. This indicates that DISK
detections consists of a small number of certain detections, with a large
number of uncertain detections, whereas DeDoDe and SiLK detections contain a
larger spectrum. This might be due to the training of DISK, which is
regularized to have single detections in patches by local softmax [32], while
DeDoDe and SiLK do not. It can also be observed that ALIKED shows very minor
gains from increasing the number of keypoints. This indicates that ALIKED may
be better suited for usecases where very few but reliable keypoints are
required, while DeDoDe scales better for larger numbers. Finally, note that
while DeDoDe, ALIKED, and DISK are trained on the MegaDepth dataset, SiLK is
trained only using synthetic homographies.

Image Matching Challenge 2022: The Image Matching Challenge 2022 [14] is
composed of challenging uncalibrated relative pose estimation pairs. Different
from MegaDepth-1500, the test set is hidden and does not derive from
MegaDepth, and may therefore better indicate generalization performance,
especially for models such as DeDoDe that train on MegaDepth. We follow the
setup in SiLK [10] and use $30000$ keypoints, and MAGSAC++ [1] with a
threshold of $0.25$ pixels. We use a fixed image size of $784\times 784$. We
follow the approach in RoMa [9] and report results on the hidden test set. We
present results in Table 4. Here too, DeDoDe achieves significant improvements
to the state-of-the-art, with results competitive with the graph neural
network (GNN) matcher SuperGlue [26], and surpassing the recent SiLK [10] with
+7.4 points mAA.

Table 4: SotA comparison on the IMC2022 benchmark. Relative pose estimation results on the IMC2022 [14] hidden test set, measured in mAA (higher is better). The top portion contains dense and detector free matching methods, the middle part keypoint matching methods, while the bottom portion contains traditional MNN matching from keypoints and descriptors. Method $\downarrow$ mAA $\rightarrow$ | $@10$
---|---
LoFTR [29] CVPR’21 | 78.3
MatchFormer [34] ACCV’22 | 78.3
QuadTree [31] ICLR’22 | 81.7
ASpanFormer [5] ECCV’22 | 83.8
RoMa [9] Arxiv’23 | 88.0
SP [6]+SuperGlue [26] | 72.4
DISK [32] Neurips’20 | 64.8
ALIKED [39] IEEE-TIM’23 | 64.9
SiLK [10] ICCV’23 | 68.5
DeDoDe-B | 72.9
DeDoDe-G | 75.8

###  6.6 Ablation Study

Design Decisions:

Table 5: Ablation study on MegaDepth-Repeatability-10k. We investigate the performance effect of two of our contributions on the detector repeatability using a fixed resolution of $512\times 512$. Method | Repeatability @ 10k
---|---
| $0.1\%$ | $0.2\%$ | $0.5\%$
I: Prior Distribution as Target | 34.8 | 61.3 | 78.0
II: No Coverage Loss | 29.7 | 59.4 | 78.0
DeDoDe (512x512) | 37.1 | 68.1 | 84.8

Here we investigate the impact of our design decisions. In Setting I, we
supervise directly using the detection prior in Section 4.3. This decreases
performance, in particular for coarse thresholds. In Setting II we do not
include the coverage term introduced in Section 4.5. This decreases
performance across all thresholds. We attribute this to the network not
learning to attenuate certainty in unmatchable regions, _e.g_., sky.

Exchanging Components with Previous Methods: In this section we compare the
effect of the detector and descriptor of DeDoDe by exchanging components of
SIFT, DISK, and DeDoDe. We present results in Table 6. Interestingly, using
SIFT detections with DeDoDe descriptions outperforms the DISK feature matcher.
Similarly, replacing the DISK descriptor with DeDoDe also improves
performance. Note however that while results are improved compared to the
original methods with DeDoDe descriptor, it is a significant decrease compared
to when using DeDoDe detections, with an absolute decrease of 8 points AUC.

Replacing the detector in DISK with DeDoDe slightly decreases the performance.
We attribute this to the training of DISK, which learns keypoints that is
likely to be mutual nearest neighbours of the descriptor. SIFT descriptors, in
contrast to DISK, require scale and rotation estimation from the keypoint. To
circumvent this issue we sample SIFT keypoints within a distance of $1$ pixel
from DeDoDe keypoints. Note that this is a inferior version to our original
detector as it results in a subset of the original SIFT keypoints. It is
therefore not surprising that results are slightly degraded.

Table 6: Ablation study on the Megadepth-1500 benchmark. Here we ablate the gains from incorporating the DeDoDe detector and descriptor to matching. As baselines we compare to using SIFT, or DISK [32]. ∗: SIFT requires scale and orientation estimation from the keypoints, see text for details. Detector/Descriptor $\downarrow$ AUC $\rightarrow$ | $@5^{\circ}$ | $@10^{\circ}$ | $@20^{\circ}$
---|---|---|---
SIFT/SIFT [19] | 36.5 | 50.2 | 62.7
DISK/DISK [32] NeurIps’20 | 35.0 | 51.4 | 64.9
DeDoDe/DeDoDe-B | 49.4 | 65.5 | 77.7
SIFT/DeDoDe-B | 41.1 | 56.8 | 69.8
DISK/DeDoDe-B | 41.5 | 57.9 | 71.0
DeDoDe/SIFT∗ | 34.1 | 49.1 | 62.0
DeDoDe/DISK | 33.1 | 48.9 | 61.7

##  7 Conclusion

We have presented DeDoDe, a descriptor-agnostic modular approach to geometric
keypoint detection. We did this by formulating keypoint detection as detecting
points that have successfully created 3D tracks in large-scale SfM. We
introduced a two-view objective, in order to reduce the sparseness of the
detections, and furthermore proposed a suitable descriptor objective. Our
experiments show that DeDoDe signficantly improves on the state-of-the-art.
DeDoDe enables a new research direction towards a modular detection and
matching framework.

Limitations:

  1. 1.

DeDoDe is dependent on the base detector. This means that potential stable
keypoints can be missed. We remedied this by introducing a top-$k$ semi-
supervised objective during training.

  2. 2.

DeDoDe is not trained with augmentations, contrary to common practice [18,
10]. It is possible that appropriate augmentation would increase the
performance/generalization of our method.

  3. 3.

DeDoDe tends to produce a large number of potential keypoints. In the few-
keypoint setting this means that the repeatability degrades, as many keypoints
are missed. Adjusting the hyperparameters of the keypoint target distribution
may be able to address these issues.

  4. 4.

The DeDoDe detector produces only keypoint locations, and does not estimate
orientation/scale. This could potentially be remedied by using a separate
network for estimating the local frame [21, 15, 35], although this would come
at an additional computational cost.

#### Acknowledgements:

We thank the reviewers for the constructive feedback that helped improve the
paper. We thank Emanuel Sanchez Aimar and Ioannis Athanasiadis for the early
discussions. This work was supported by the Wallenberg Artificial
Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut
and Alice Wallenberg Foundation; and by the strategic research environment
ELLIIT funded by the Swedish government. The computational resources were
provided by the National Academic Infrastructure for Supercomputing in Sweden
(NAISS) at C3SE partially funded by the Swedish Research Council through grant
agreement no. 2022-06725, and by the Berzelius resource, provided by the Knut
and Alice Wallenberg Foundation at the National Supercomputer Centre.

## References

  * Barath et al. [2020] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas.  Magsac++, a fast, reliable and accurate robust estimator.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1304–1312, 2020.
  * Barroso-Laguna et al. [2019] Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, and Krystian Mikolajczyk.  Key. net: Keypoint detection by handcrafted and learned cnn filters.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 5836–5844, 2019.
  * Bökman and Kahl [2022] Georg Bökman and Fredrik Kahl.  A case for using rotation invariant features in state of the art feature matchers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5110–5119, 2022.
  * Chen et al. [2021] Hongkai Chen, Zixin Luo, Jiahui Zhang, Lei Zhou, Xuyang Bai, Zeyu Hu, Chiew-Lan Tai, and Long Quan.  Learning to match features with seeded graph matching network.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6301–6310, 2021.
  * Chen et al. [2022] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan.  ASpanFormer: Detector-Free Image Matching with Adaptive Span Transformer.  In _Proc. European Conference on Computer Vision (ECCV)_ , 2022.
  * DeTone et al. [2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 224–236, 2018.
  * Dusmanu et al. [2019] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  In _Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Edstedt et al. [2023a] Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg.  DKM: Dense Kernelized Feature Matching for Geometry Estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17765–17775, 2023a.
  * Edstedt et al. [2023b] Johan Edstedt, Qiyu Sun, Georg Bökman, Mårten Wadenbäck, and Michael Felsberg.  RoMa: Revisiting robust lossses for dense feature matching.  _arXiv preprint arXiv:2305.15404_ , 2023b.
  * Gleize et al. [2023] Pierre Gleize, Weiyao Wang, and Matt Feiszli.  SiLK: Simple Learned Keypoints.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Harris et al. [1988] Chris Harris, Mike Stephens, et al.  A combined corner and edge detector.  In _Proceedings of the Alvey vision conference_ , page 147–151, 1988.
  * Hartmann et al. [2014] Wilfried Hartmann, Michal Havlena, and Konrad Schindler.  Predicting matchability.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 9–16, 2014.
  * Howard et al. [2022] Addison Howard, Eduard Trulls, etru1927, Kwang Moo Yi, old ufo, Sohier Dane, and Yuhe Jin.  Image matching challenge 2022, 2022.
  * Lee et al. [2021] Jongmin Lee, Yoonwoo Jeong, and Minsu Cho.  Self-supervised learning of image scale and orientation.  In _31st British Machine Vision Conference 2021, BMVC 2021, Virtual Event, UK_. BMVA Press, 2021.
  * Li et al. [2022] Kunhong Li, Longguang Wang, Li Liu, Qing Ran, Kai Xu, and Yulan Guo.  Decoupling makes weakly supervised local feature better.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15838–15848, 2022.
  * Li and Snavely [2018] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2041–2050, 2018.
  * Lindenberger et al. [2023] Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys.  LightGlue: Local Feature Matching at Light Speed.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Lowe [2004] David G Lowe.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ , 60(2):91–110, 2004.
  * Mishchuk et al. [2017] Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas.  Working hard to know your neighbor’s margins: Local descriptor learning loss.  _Advances in neural information processing systems_ , 30, 2017.
  * Mishkin et al. [2018] Dmytro Mishkin, Filip Radenovic, and Jiri Matas.  Repeatability is not enough: Learning affine regions via discriminability.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 284–300, 2018.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  DINOv2: Learning robust visual features without supervision.  _arXiv:2304.07193_ , 2023.
  * Pautrat et al. [2021] Rémi Pautrat, Juan-Ting Lin, Viktor Larsson, Martin R Oswald, and Marc Pollefeys.  SOLD2: Self-supervised occlusion-aware line description and detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11368–11378, 2021.
  * Pautrat et al. [2023] Rémi Pautrat, Daniel Barath, Viktor Larsson, Martin R Oswald, and Marc Pollefeys.  DeepLSD: Line segment detection and refinement with deep image gradients.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17327–17336, 2023.
  * Revaud et al. [2019] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel.  R2d2: Reliable and repeatable detector and descriptor.  _Advances in neural information processing systems_ , 32:12405–12415, 2019.
  * Sarlin et al. [2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4938–4947, 2020.
  * Sattler et al. [2018] Torsten Sattler, Will Maddern, Carl Toft, Akihiko Torii, Lars Hammarstrand, Erik Stenborg, Daniel Safari, Masatoshi Okutomi, Marc Pollefeys, Josef Sivic, et al.  Benchmarking 6dof outdoor visual localization in changing conditions.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8601–8610, 2018.
  * Strecha et al. [2009] Christoph Strecha, Albrecht Lindner, Karim Ali, and Pascal Fua.  Training for task specific keypoint detection.  In _Pattern Recognition: 31st DAGM Symposium, Jena, Germany, September 9-11, 2009. Proceedings 31_ , pages 151–160. Springer, 2009.
  * Sun et al. [2021] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8922–8931, 2021.
  * Taira et al. [2018] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7199–7209, 2018.
  * Tang et al. [2022] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan.  Quadtree attention for vision transformers.  In _International Conference on Learning Representations_ , 2022.
  * Tyszkiewicz et al. [2020] Michal J. Tyszkiewicz, Pascal Fua, and Eduard Trulls.  DISK: learning local features with policy gradient.  In _NeurIPS_ , 2020.
  * Verdie et al. [2015] Yannick Verdie, Kwang Yi, Pascal Fua, and Vincent Lepetit.  Tilde: A temporally invariant learned detector.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5279–5288, 2015.
  * Wang et al. [2022] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen.  MatchFormer: Interleaving attention in transformers for feature matching.  In _Asian Conference on Computer Vision_ , 2022.
  * Yan et al. [2022] Pei Yan, Yihua Tan, Shengzhou Xiong, Yuan Tai, and Yansheng Li.  Learning soft estimator of keypoint scale and orientation with probabilistic covariant loss.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19406–19415, 2022.
  * Yi et al. [2016] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In _European conference on computer vision_ , pages 467–483. Springer, 2016.
  * Yu et al. [2023] Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Jiyang Yu, and Wu Feng.  ASTR: Adaptive spot-guided transformer for consistent local feature matching.  In _The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR)_ , 2023.
  * Zhao et al. [2022] Xiaoming Zhao, Xingming Wu, Jinyu Miao, Weihai Chen, Peter CY Chen, and Zhengguo Li.  Alike: Accurate and lightweight keypoint detection and descriptor extraction.  _IEEE Transactions on Multimedia_ , 2022.
  * Zhao et al. [2023] Xiaoming Zhao, Xingming Wu, Weihai Chen, Peter C. Y. Chen, Qingsong Xu, and Zhengguo Li.  Aliked: A lighter keypoint and descriptor extraction network via deformable transformation.  _IEEE Transactions on Instrumentation & Measurement_, 72:1–16, 2023.
  * Zhu and Liu [2023] Shengjie Zhu and Xiaoming Liu.  PMatch: Paired masked image modeling for dense geometric matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.

Supplementary Material

In this supplementary material we present implementation details, qualitative
examples, and theory that could not fit in the main text.

##  Appendix A Definitions of Evaluation Metrics

AUC: We evaluate performance using the Area Under the Curve (AUC). We define
the AUC$@t$ as the integral of the precision up to the threshold $t$. In
practice, this is approximated using the trapezoidal rule, as in previous work
[29, 8].

Precision: The relative pose estimation precision is defined as

${\rm Precision}@t=\max(|\varepsilon_{\hat{t}}|,|\varepsilon_{\hat{R}}|)<t$
(16)

where

$\varepsilon_{\hat{t}}=\arccos\frac{\langle t,\hat{t}\rangle}{\lVert
t\rVert\cdot\lVert\hat{t}\rVert}$ (17)

and

$\varepsilon_{\hat{R}}=\arccos\big{(}({\rm tr}(\hat{R}^{\top}R)-1)/2\big{)}.$
(18)

Repeatability: First, let

$\textbf{x}^{\mathcal{A}}_{\rm kp\cap
W}=\\{x^{\mathcal{A}}_{i}|x^{\mathcal{A}}_{i}\in{\rm
dom}(W)\cap\textbf{x}^{\mathcal{A}}_{\rm kp}\\},$ (19)

where W is the ground truth warp, and its domain dom(W) are the pixels where
it is well defined. In the case of MegaDepth, we will only consider keypoints
in the MVS regions. We define repeatability as

$\frac{1}{|\textbf{x}^{\mathcal{A}}_{\rm kp\cap
W}|}\sum_{x^{\mathcal{A}}_{i}\in\textbf{x}^{\mathcal{A}}_{\rm kp\cap
W}}\bigg{(}\min_{x^{\mathcal{B}}_{i}\in\textbf{x}^{\mathcal{B}}_{\rm
kp}}\lVert x^{\mathcal{B}}_{i}-W(x^{\mathcal{A}}_{i})\rVert_{2}\bigg{)}<t.$
(20)

In words, this measures the the percentage of keypoints detections in the MVS
covisible region with a corresponding keypoint within a certain threshold in
the other image.

mAA: For IMC-2022 the performance is measured in mAA, which closely resembles
AUC. We further detail the metric below. Given a an estimated fundamental
matrix, the error is computed in terms of rotation ($\varepsilon_{R}$ in
degrees) and translation ($\varepsilon_{T}$ in meters). Given one threshold
over each, the pose is classified as accurate if it meets both thresholds. In
code:
thresholds_r = np.linspace(1, 10, 10) # In degrees.
thresholds_t = np.geomspace(0.2, 5, 10) # In meters.
The percentage of image pairs that meet every pair of thresholds is computed,
the results are averaged over all thresholds, which rewards more accurate
poses. As the dataset contains multiple scenes, which may have a different
number of pairs, the metric is computed separately for each scene and then
averaged. We refer to this metric as mAA$@10$.

##  Appendix B DeDoDe Further Implementation Details

In the following paragraphs, we describe implementation details that did not
fit in the main text.

Detector Objective Further Details: We set the prior Uniform constant $C$ such
that the peak of the Gaussian centered around the deltas is a factor
$\exp(50)$ larger compared to the uniform. In practice, we do these
computations in log-space. This ensures that all consistent keypoints are
detected, while still allowing the network to detect additional keypoints in
cases where the target distribution is overly sparse. We found that the
performance of the network was robust to the precise value of $C$.

Further Detector Architecture Details: For DeDoDe detector dimensionality of
the encoded context is $[32,128,256]$ for strides $[2,4,8]$.

Descriptor Objective Further Details: In addition to enforcing the keypoints
to be mutual nearest neighbours, we enforce that the distance be smaller than
$0.5\%$ of the image in both directions. We did not find that this had a major
impact on performance, possibly due to very few mutual nearest neighbours not
fulfilling the criteria.

Descriptor Architecture Further Details: For DeDoDe descriptor-B
dimensionality of the encoded context is $[32,128,256]$ for strides $[2,4,8]$
respectively, for DeDoDe descriptor-G we additionally encode context of
dimension $512$ for the DINOv2 [22] features at stride $14$.

Training Further Details: We use a base learning rate of $2\cdot 10^{-5}$ for
the encoders and $1\cdot 10^{-4}$ for the decoders. We use a cosine decay
learning rate scheduler.

##  Appendix C Runtime:

Here we measure the runtime of DeDoDe, and compare it to DISK [32]. We run
both methods at the inference resolution used in the SotA experiments, which
is $1024\times 1024$ for DISK and $784\times 784$ for DeDoDe. We present
results in Table 7. We find that DeDoDe has a similar runtime to DISK, and is
about $\sim 2.5\times$ faster for keypoint-only extraction.

| Time (ms)
---|---
DeDoDe Det | 62.5
DeDoDe Det+Desc-B | 103.4
DeDoDe Det+Desc-G | 160.0
DISK Det+Desc | 170.0
Table 7: Inference time comparison. We compare the inference time of DeDode to
DISK.

##  Appendix D Inference Settings for Detector+Descriptors

DeDoDe: We use a resolution of $784\times 784$ We use a dual-softmax threshold
of 0.01.

DISK: We follow the suggested inference approach, and use images of size
$1024\times 1024$, with a NMS radius of 5. We use a dual-softmax matcher with
confidence threshold $0.01$. We additionally investigated the ratio-test but
found it to give inferior results on IMC22.

SiLK: We use the  official settings of SiLK for IMC-2022, where the longer
side of the image is resized to $720$, a dual-softmax matcher with confidence
threshold of $0.01$, and use $30000$ keypoints. Using this threshold, SiLK
produces many outliers, as can be seen in Figure 6. For a fair comparison on
MegaDepth-1500 we therefore instead use a threshold of $0.05$, which we found
beneficial for SiLK.

Figure 6: Issues with SiLK confidence. At tight confidence thresholds SiLK
tends to produce too few but accurate correspondences. However, at looser
thresholds a large number of outlier correspondences arise.

ALIKED: We find ALIKED to perform the best using a resolution of $800\times
800$, we sample $10000$ keypoints per image, and use a dual-softmax matcher
with confidence threshold of $0.01$.

Figure 7: Top-10k Keypoints of DeDoDe. In contrast to previous methods, our
method is able to produce a larger amount of repeatable keypoints, leading to
more accurate estimation results. Best viewed in high-resolution.

##  Appendix E Robustness to Large Rotations

We revisit the pair in Figure 5 in a more difficult setting by rotating the
first image by $180^{\circ}$. We find that despite not being trained on large
rotations, DeDoDe produces good matches. We leave investigations of how to
improve performance on non-upright images to future work.

Figure 8: Robustness of DeDoDe to large rotation. Despite not being trained on
large rotations, and not having an equivariant network architecture, DeDoDe is
able to match under even extreme changes in rotation. Rotations do however
produce a larger number of outlier correspondences as can be seen in the
figure.

Figure 9: DeDoDe matches on IMC-2022. We show matches of two publicly
available image pairs from the IMC2022 benchmark. Figure 10: DeDoDe matches on
the MegaDepth-1500 benchmark.

##  Appendix F Why is top-$k$ Needed?

In Section 4.5 we argued that the top-k thresholding is needed to stabilize
training. Here we provide some theoretical reasons why the non-thresholded
objective has issues.

To see this, note that the cross-entropy is 0 if and only if
$p_{f_{\theta}}=\delta_{x_{i}}(x)$, for any $x_{i}$. Hence this objective has
multiple degenerate solutions. We found that when training, the network
collapses into predicting a single peak.

One could instead consider a Kullback-Leibler (KL) divergence between the
distributions. This divergence is however also minimized by any
$p_{f_{\theta}}=\delta_{x_{i}}(x)$.

Stability issues in self-supervised objectives are well-known and many
solutions have been proposed, see _e.g_., Grill et al. [11], Oquab et al.
[22].

Our proposed top-$k$ objective is a simple way of avoiding these
instabilities. The binarization encourages the network to predict a flat
distribution over certain keypoints, ensuring that it does not collapse. The
downside of the objective is, as discussed in Section 4.5 that $k$ is an
additional hyperparameter that is not obvious how to set optimally and will
depend on the dataset.

##  Appendix G Additional Qualitative Examples

In Figure 9 we provide qualitative examples of DeDoDe on the publically
released images of the IMC-2022 benchmark. In Figure 10 we provide an
additional example of DeDoDe on the MegaDepth-1500 benchmark.