# RoMa: Robust Dense Feature Matching

Johan Edstedt1 Qiyu Sun2 Georg Bökman3 Mårten Wadenbäck1 Michael Felsberg1
1Linköping University, 2East China University of Science and Technology,
3Chalmers University of Technology

###### Abstract

Feature matching is an important computer vision task that involves estimating
correspondences between two images of a 3D scene, and dense methods estimate
all such correspondences. The aim is to learn a robust model, __i.e_.
missing_, a model able to match under challenging real-world changes. In this
work, we propose such a model, leveraging frozen pretrained features from the
foundation model DINOv2. Although these features are significantly more robust
than local features trained from scratch, they are inherently coarse. We
therefore combine them with specialized ConvNet fine features, creating a
precisely localizable feature pyramid. To further improve robustness, we
propose a tailored transformer match decoder that predicts anchor
probabilities, which enables it to express multimodality. Finally, we propose
an improved loss formulation through regression-by-classification with
subsequent robust regression. We conduct a comprehensive set of experiments
that show that our method, RoMa, achieves significant gains, setting a new
state-of-the-art. In particular, we achieve a 36% improvement on the extremely
challenging WxBS benchmark. Code is provided at github.com/Parskatt/RoMa.

Figure 1: RoMa is robust, _i.e_., able to match under extreme changes. We
propose RoMa, a model for dense feature matching that is robust to a wide
variety of challenging real-world changes in scale, illumination, viewpoint,
and texture. We show correspondences estimated by RoMa on the extremely
challenging benchmark WxBS [35], where most previous methods fail, and on
which we set a new state-of-the-art with an improvement of 36% mAA. The
estimated correspondences are visualized by grid sampling coordinates
bilinearly from the other image, using the estimated warp, and multiplying
with the estimated confidence.

##  1 Introduction

Figure 2: Illustration of our robust approach RoMa. Our contributions are
shown with green highlighting and a checkmark, while previous approaches are
indicated with gray highlights and a cross. Our first contribution is using a
frozen foundation model for coarse features, compared to fine-tuning or
training from scratch. DINOv2 lacks fine features, which are needed for
accurate correspondences. To tackle this, we combine the DINOv2 coarse
features with specialized fine features from a ConvNet, see Section 3.2.
Second, we propose an improved coarse match decoder $D_{\theta}$, which
typically is a ConvNet, with a coordinate agnostic Transformer decoder that
predicts anchor probabilities instead of directly regressing coordinates, see
Section 3.3. Third, we revisit the loss functions used for dense feature
matching. We argue from a theoretical model that the global matching stage
needs to model multimodal distributions, and hence use a regression-by-
classification loss instead of an L2 loss. For the refinement, we in contrast
use a robust regression loss, as the matching distribution is locally
unimodal. These losses are further discussed in Section 3.4. The impact of our
contributions is ablated in our extensive ablation study in Table 2.

Feature matching is the computer vision task of from two images estimating
pixel pairs that correspond to the same 3D point. It is crucial for downstream
tasks such as 3D reconstruction [43] and visual localization [40]. Dense
feature matching methods [49, 52, 17, 36] aim to find all matching pixel-pairs
between the images. These dense methods employ a coarse-to-fine approach,
whereby matches are first predicted at a coarse level and successively refined
at finer resolutions. Previous methods commonly learn coarse features using 3D
supervision [41, 44, 52, 17]. While this allows for specialized coarse
features, it comes with downsides. In particular, since collecting real-world
3D datasets is expensive, the amount of available data is limited, which means
models risk overfitting to the training set. This in turn limits the models
robustness to scenes that differ significantly from what has been seen during
training. A well-known approach to limit overfitting is to freeze the backbone
used [47, 54, 29]. However, using frozen backbones pretrained on ImageNet
classification, the out-of-the-box performance is insufficient for feature
matching (see experiments in Table 1). A recent promising direction for frozen
pretrained features is large-scale self-supervised pretraining using Masked
image Modeling (MIM) [24, 56, 62, 37]. The methods, including DINOv2 [60],
retain local information better than classification pretraining [60] and have
been shown to generate features that generalize well to dense vision tasks.
However, the application of DINOv2 in dense feature matching is still
complicated due to the lack of fine features, which are needed for refinement.

We overcome this issue by leveraging a frozen DINOv2 encoder for coarse
features, while using a proposed specialized ConvNet encoder for the fine
features. This has the benefit of incorporating the excellent general features
from DINOv2, while simultaneuously having highly precise fine features. We
find that features specialized for only coarse matching or refinement
significantly outperform features trained for both tasks jointly. These
contributions are presented in more detail in Section 3.2. We additionally
propose a Transformer match decoder that while also increasing performance for
the baseline, particularly improves performance when used to predict anchor
probabilities instead of regressing coordinates in conjunction with the DINOv2
coarse encoder. This contribution is elaborated further in Section 3.3.

Lastly, we investigate how to best train dense feature matchers. Recent SotA
dense methods such as DKM [17] use a non-robust regression loss for the coarse
matching as well as for the refinement. We argue that this is not optimal as
the matching distribution at the coarse stage is often multimodal, while the
conditional refinement is more likely to be unimodal. Hence requiring
different approaches to training. We motivate this from a theoretical
framework in Section 3.4. Our framework motivates a division of the coarse and
fine losses into seperate paradigms, regression-by-classification for the
global matches using coarse features, and robust regression for the refinement
using fine features.

Our full approach, which we call RoMa, is robust to extremely challenging
real-world cases, as we demonstrate in Figure 1. We illustrate our approach
schematically in Figure 2. In summary, our contributions are as follows:

  1. (a)

We integrate frozen features from the foundation model DINOv2 [37] for dense
feature matching. We combine the coarse features from DINOv2 with specialized
fine features from a ConvNet to produce a precisely localizable yet robust
feature pyramid. See Section 3.2.

  2. (b)

We propose a Transformer-based match decoder, which predicts anchor
probabilities instead of coordinates. See Section 3.3.

  3. (c)

We improve the loss formulation. In particular, we use a regression-by-
classification loss for coarse global matches, while we use robust regression
loss for the refinement stage, both of which we motivate from a theoretical
analysis. See Section 3.4.

  4. (d)

We conduct an extensive ablation study over our contributions, and SotA
experiments on a set of diverse and competitive benchmarks, and find that RoMa
sets a new state-of-the-art. In particular, achieving a gain of 36% on the
difficult WxBS benchmark. See Section 4.

##  2 Related Work

###  2.1 Sparse $\to$ Detector Free $\to$ Dense Matching

Feature matching has traditionally been approached by keypoint detection and
description followed by matching the descriptions [33, 4, 14, 39, 41, 53].
Recently, the detector-free approach [44, 7, 46, 12] replaces the keypoint
detection with dense matching on a coarse scale, followed by mutual nearest
neighbors extraction, which is followed by refinement. The dense approach [34,
50, 51, 17, 63, 36] instead estimates a dense warp, aiming to estimate every
matchable pixel pair.

###  2.2 Self-Supervised Vision Models

Inspired by language Transformers [15] foundation models [8] pre-trained on
large quantities of data have recently demonstrated significant potential in
learning all-purpose features for various visual models via self-supervised
learning. Caron et al. [11] observe that self-supervised ViT features capture
more distinct information than supervised models do, which is demonstrated
through label-free self-distillation. iBOT [62] explores MIM within a self-
distillation framework to develop a semantically rich visual tokenizer,
yielding robust features effective in various dense downstream tasks. DINOv2
[37] reveals that self-supervised methods can produce all-purpose visual
features that work across various image distributions and tasks after being
trained on sufficient datasets without finetuning.

###  2.3 Robust Loss Formulations

Robust Regression Losses: Robust loss functions provide a continuous
transition between an inlier distribution (typically highly concentrated), and
an outlier distribution (wide and flat). Robust losses have, _e.g_., been used
as regularizers for optical flow [5, 6], robust smoothing [18], and as loss
functions [3, 32].

Regression by Classification: Regression by classification [57, 58, 48]
involves casting regression problems as classification by, _e.g_., binning.
This is particularly useful for regression problems with sharp borders in
motion, such as stereo disparity [19, 22]. Germain et al. [20] use a
regression-by-classification loss for absolute pose regression.

Classification then Regression: Li et al. [27], and Budvytis et al. [9]
proposed hierarchical classification-regression frameworks for visual
localization. Sun et al. [44] optimize the model log-likelihood of mutual
nearest neighbors, followed by L2 regression-based refinement for feature
matching.

##  3 Method

In this section, we detail our method. We begin with preliminaries and
notation for dense feature matching in Section 3.1. We then discuss our
incorporation of DINOv2 [37] as a coarse encoder, and specialized fine
features in Section 3.2. We present our proposed Transformer match decoder in
Section 3.3. Finally, our proposed loss formulation in Section 3.4. A summary
and visualization of our full approach is provided in Figure 2. Further
details on the exact architecture are given in the supplementary.

###  3.1 Preliminaries on Dense Feature Matching

Dense feature matching is, given two images $I^{\mathcal{A}},I^{\mathcal{B}}$,
to estimate a dense warp $W^{\mathcal{A}\to\mathcal{B}}$ (mapping coordinates
$x^{\mathcal{A}}$ from $I^{\mathcal{A}}$ to $x^{\mathcal{B}}$ in
$I^{\mathcal{B}}$), and a matchability score $p(x^{\mathcal{A}})$111This is
denoted as $p^{\mathcal{A}\to\mathcal{B}}$ by Edstedt et al. [17]. We omit the
$\mathcal{B}$ to avoid confusion with the conditional. for each pixel. From a
probabilistic perspective,
$p(W^{\mathcal{A}\to\mathcal{B}})=p(x^{\mathcal{B}}|x^{\mathcal{A}})$ is the
conditional matching distribution. Multiplying
$p(x^{\mathcal{B}}|x^{\mathcal{A}})p(x^{\mathcal{A}})$ yields the joint
distribution. We denote the model distribution as
$p_{\theta}(x^{\mathcal{A}},x^{\mathcal{B}})=p_{\theta}(x^{\mathcal{B}}|x^{\mathcal{A}})p_{\theta}(x^{\mathcal{A}})$.
When working with warps, _i.e_., where
$p_{\theta}(x^{\mathcal{B}}|x^{\mathcal{A}})$ has been converted to a
deterministic mapping, we denote the model warp as
$\hat{W}^{\mathcal{A}\to\mathcal{B}}$. Viewing the predictive distribution as
a warp is natural in high resolution, as it can then be seen as a
deterministic mapping. However, due to multimodality, it is more natural to
view it in the probabilistic sense at coarse scales.

The end goal is to obtain a good estimate over correspondences of coordinates
$x^{\mathcal{A}}$ in image $I^{\mathcal{A}}$ and coordinates $x^{\mathcal{B}}$
in image $I^{\mathcal{B}}$. For dense feature matchers, estimation of these
correspondences is typically done by a one-shot coarse _global matching_ stage
(using coarse features) followed by subsequent _refinement_ of the estimated
warp and confidence (using fine features).

We use the recent SotA dense feature matching model DKM [17] as our baseline.
For consistency, we adapt the terminology used there. We denote the coarse
features used to estimate the initial warp, and the fine features used to
refine the warp by

$\\{\varphi^{\mathcal{A}}_{\text{coarse}},\varphi^{\mathcal{A}}_{\text{fine}}\\}=F_{\theta}(I^{\mathcal{A}}),\\{\varphi^{\mathcal{B}}_{\text{coarse}},\varphi^{\mathcal{B}}_{\text{fine}}\\}=F_{\theta}(I^{\mathcal{B}}),$
(1)

where $F_{\theta}$ is a neural network feature encoder. We will leverage
DINOv2 for extraction of
$\varphi^{\mathcal{A}}_{\text{coarse}},\varphi^{\mathcal{B}}_{\text{coarse}}$,
however, DINOv2 features are not precisely localizable, which we tackle by
combining the coarse features with precise local features from a specialized
ConvNet backbone. See Section 3.2 for details.

The coarse features are matched with global matcher $G_{\theta}$ consisting of
a match encoder $E_{\theta}$ and match decoder $D_{\theta}$,

$\left\\{\begin{aligned}
\big{(}\hat{\text{W}}^{\mathcal{A}\to\mathcal{B}}_{\text{coarse}},\enspace
p^{\mathcal{A}}_{\theta,\text{coarse}}\big{)}&=G_{\theta}(\varphi^{\mathcal{A}}_{\text{coarse}},\varphi^{\mathcal{B}}_{\text{coarse}}),\enspace\\\
G_{\theta}({\varphi}^{\mathcal{A}}_{\text{coarse}},{\varphi}^{\mathcal{B}}_{\text{coarse}})&=D_{\theta}\big{(}E_{\theta}({\varphi}^{\mathcal{A}}_{\text{coarse}},{\varphi}^{\mathcal{B}}_{\text{coarse}})\big{)}.\end{aligned}\right.$
(2)

We use a Gaussian Process [38] as the match encoder $E_{\theta}$ as in
previous work [17]. However, while our baseline uses a ConvNet to decode the
matches, we propose a Transformer match decoder $D_{\theta}$ that predicts
anchor probabilities instead of directly regressing the warp. This match
decoder is particularly beneficial in our final approach (see Table 2). We
describe our proposed match decoder in Section 3.3. The refinement of the
coarse warp $\hat{\text{W}}^{\mathcal{A}\to\mathcal{B}}_{\text{coarse}}$ is
done by the refiners $R_{\theta}$,

$\big{(}\hat{\text{W}}^{\mathcal{A}\to\mathcal{B}},\,p_{\theta}^{\mathcal{A}}\big{)}=R_{\theta}\big{(}\varphi_{\text{fine}}^{\mathcal{A}},\varphi_{\text{fine}}^{\mathcal{B}},\hat{\text{W}}^{\mathcal{A}\to\mathcal{B}}_{\text{coarse}},p^{\mathcal{A}}_{\theta,\text{coarse}}\big{)}.$
(3)

As in previous work, the refiner is composed of a sequence of ConvNets (using
strides $\\{1,2,4,8\\}$) and can be decomposed recursively as

$\big{(}\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i},\;p_{i,\theta}^{\mathcal{A}}\big{)}=R_{\theta,i}(\varphi_{i}^{\mathcal{A}},\varphi_{i}^{\mathcal{B}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1},p_{\theta,i+1}^{\mathcal{A}}),$
(4)

where the stride is $2^{i}$. The refiners predict a residual offset for the
estimated warp, and a logit offset for the certainty. As in the baseline they
are conditioned on the outputs of the previous refiner by using the previously
estimated warp to a) stack feature maps from the images, and b) construct a
local correlation volume around the previous target.

The process is repeated until reaching full resolution. We use the same
architecture as in the baseline. Following DKM, we detach the gradients
between the refiners and upsample the warp bilinearly to match the resolution
of the finer stride.

Probabilistic Notation: When later defining our loss functions, it will be
convenient to refer to the outputs of the different modules in a probabilistic
notation. We therefore introduce this notation here first for clarity.

We denote the probability distribution modeled by the global matcher as

$p_{\theta}(x_{\text{coarse}}^{\mathcal{A}},x_{\text{coarse}}^{\mathcal{B}})=G_{\theta}(\varphi_{\text{coarse}}^{\mathcal{A}},\varphi_{\text{coarse}}^{\mathcal{B}}).$
(5)

Here we have dropped the explicit dependency on the features and the previous
estimate of the marginal for notational brevity. Note that the output of the
global matcher will sometimes be considered as a discretized distribution
using anchors, or as a decoded warp. We do not use separate notation for these
two different cases to keep the notation uncluttered.

We denote the probability distribution modeled by a refiner at scale
$s=c2^{i}$ as

$p_{\theta}(x_{i}^{\mathcal{A}},x_{i}^{\mathcal{B}}|\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})=R_{\theta,i}(\varphi_{i}^{\mathcal{A}},\varphi_{i}^{\mathcal{B}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1},p_{\theta,i+1}^{\mathcal{A}}),$
(6)

The basecase $\hat{W}^{\mathcal{A}\to\mathcal{B}}_{\text{coarse}}$ is computed
by decoding
$p_{\theta}(x_{\text{coarse}}^{\mathcal{B}}|x_{\text{coarse}}^{\mathcal{A}})$.
As for the global matcher we drop the explicit dependency on the features.

###  3.2 Robust and Localizable Features

We first investigate the robustness of DINOv2 to viewpoint and illumination
changes compared to VGG19 and ResNet50 on the MegaDepth [28] dataset. To
decouple the backbone from the matching model we train a single linear layer
on top of the frozen model followed by a kernel nearest neighbour matcher for
each method. We measure the performance both in average end-point-error (EPE)
on a standardized resolution of 448$\times$448, and by what we call the
Robustness $\%$ which we define as the percentage of matches with an error
lower than $32$ pixels. We refer to this as robustness, as, while these
matches are not necessarily accurate, it is typically sufficient for the
refinement stage to produce a correct adjustment.

We present results in Table 1. We find that DINOv2 features are significantly
more robust to changes in viewpoint than both ResNet and VGG19. Interestingly,
we find that the VGG19 features are worse than the ResNet features for coarse
matching, despite VGG feature being widely used as local features [16, 42,
52]. Further details of this experiment are provided in the supplementary
material.

Table 1: Evaluation of frozen features on MegaDepth. We compare the VGG19 and ResNet50 backbones commonly used in feature matching with the generalist features of DINOv2. Method |  EPE $\downarrow$ |  Robustness % $\uparrow$
---|---|---
VGG19 | 87.6 | 43.2
RN50 | 60.2 | 57.5
DINOv2 | 27.1 | 85.6

In DKM [17], the feature encoder $F_{\theta}$ is assumed to consist of a
single network producing a feature pyramid of coarse and fine features used
for global matching and refinement respectively. This is problematic when
using DINOv2 features as only features of stride 14 exist. We therefore
decouple $F_{\theta}$ into
$\\{F_{\text{coarse},\theta},F_{\text{fine},\theta}\\}$ and set
$F_{\text{coarse},\theta}=\text{DINOv2}$. The coarse features are extracted as

$\varphi^{\mathcal{A}}_{\text{coarse}}=F_{\text{coarse},\theta}(I^{\mathcal{A}}),\varphi^{\mathcal{B}}_{\text{coarse}}=F_{\text{coarse},\theta}(I^{\mathcal{B}}).$
(7)

We keep the DINOv2 encoder frozen throughout training. This has two benefits.
The main benefit is that keeping the representations fixed reduces overfitting
to the training set, enabling RoMa to be more robust. It is also additionally
significantly cheaper computationally and requires less memory. However,
DINOv2 cannot provide fine features. Hence a choice of
$F_{\text{fine},\theta}$ is needed. While the same encoder for fine features
as in DKM could be chosen, _i.e_., a ResNet50 (RN50) [23], it turns out that
this is not optimal.

We begin by investigating what happens by simply decoupling the coarse and
fine feature encoder, _i.e_., not sharing weights between the coarse and fine
encoder (even when using the same network). We find that, as supported by
Setup II in Table 2, this significantly increases performance. This is due to
the feature extractor being able to specialize in the respective tasks, and
hence call this _specialization_.

This raises a question, VGG19 features, while less suited for coarse matching
(see Table 1), could be better suited for fine localized features. We
investigate this by setting $F_{\text{fine},\theta}=\text{VGG19}$ in Setup III
in Table 2. Interestingly, even though VGG19 coarse features are significantly
worse than RN50, we find that they significantly outperform the RN50 features
when leveraged as fine features. Our finding indicates that there is an
inherent tension between fine localizability and coarse robustness. We thus
use VGG19 fine features in our full approach.

###  3.3 Transformer Match Decoder $D_{\theta}$

Regression-by-Classification: We propose to use the _regression by
classification_ formulation for the match decoder, whereby we discretize the
output space. We choose the following formulation,

$p_{\text{coarse},\theta}(x^{\mathcal{B}}|x^{\mathcal{A}})=\sum_{k=1}^{K}\pi_{k}(x^{\mathcal{A}})\mathcal{B}_{m_{k}},$
(8)

where $K$ is the quantization level, $\pi_{k}$ are the probabilities for each
component, $\mathcal{B}$ is some 2D base distribution, and
$\\{m_{k}\\}_{1}^{K}$ are anchor coordinates. In practice, we used $K=64\times
64$ classification anchors positioned uniformly as a tight cover of the image
grid, and $\mathcal{B}=\mathcal{U}$, _i.e_., a uniform distribution222This
ensures that there is no overlap between anchors and no holes in the cover..
We denote the probability of an anchor as $\pi_{k}$ and its associated
coordinate on the grid as $m_{k}$.

For refinement, the conditional is converted to a deterministic warp per
pixel. We decode the warp by argmax over the classification anchors,
$k^{*}(x)=\operatorname*{argmax}_{k}\pi_{k}(x)$, followed by a local
adjustment which can be seen as a local softargmax. Mathematically,

$\displaystyle\text{ToWarp}(p_{\text{coarse},\theta}(x^{\mathcal{B}}_{\text{coarse}}|x^{\mathcal{A}}_{\text{coarse}}))=$
$\displaystyle\frac{\sum_{i\in
N_{4}(k^{*}(x^{\mathcal{A}}_{\text{coarse}}))}\pi_{i}m_{i}}{\sum_{i\in
N_{4}(k^{*}(x^{\mathcal{A}}_{\text{coarse}}))}\pi_{i}}=\hat{W}^{\mathcal{A}\to\mathcal{B}}_{\text{coarse}},$
(9)

where $N_{4}(k^{*})$ denotes the set of $k^{*}$ and the four closest anchors
on the left, right, top, and bottom. We conduct an ablation on the Transformer
match decoder in Table 2, and find that it particularly improves results in
our full approach, using the loss formulation we propose in Section 3.4.

Decoder Architecture: In early experiments, we found that ConvNet coarse match
decoders overfit to the training resolution. Additionally, they tend to be
over-reliant on locality. While locality is a powerful cue for refinement, it
leads to oversmoothing for the coarse warp. To address this, we propose a
transformer decoder without using position encodings. By restricting the model
to only propagate by feature similarity, we found that the model became
significantly more robust.

The proposed Transformer matcher decoder consists of 5 ViT blocks, with 8
heads, hidden size D 1024, and MLP size 4096. The input is the concatenation
of projected DINOv2 [37] features of dimension 512, and the 512-dimensional
output of the GP module, which corresponds to the match encoder $E_{\theta}$
proposed in DKM [17]. The output is a vector of $B\times H\times W\times(K+1)$
where $K$ is the number of classification anchors333When used for regression,
$K$ is set to $K=2$, and the decoding to a warp is the identity function.
(parameterizing the conditional distribution
$p(x^{\mathcal{B}}|x^{\mathcal{A}})$), and the extra 1 is the matchability
score $p^{\mathcal{A}}(x^{\mathcal{A}})$ .

###  3.4 Robust Loss Formulation

Intuition: The conditional match distribution at coarse scales is more likely
to exhibit multimodality than during refinement, which is conditional on the
previous warp. This means that the coarse matcher needs to model multimodal
distributions, which motivates our regression-by-classification approach. In
contrast, the refinement of the warp needs only to represent unimodal
distributions, which motivates our robust regression loss.

Theoretical Model:

Figure 3: Illustration of localizability of matches. At infinite resolution
the match distribution can be seen as a 2D surface (illustrated as 1D lines in
the figure), however at a coarser scale $s$ this distribution becomes blurred
due to motion boundaries. This means it is necessary to both use a model and
an objective function capable of representing multimodal distributions.

We model the matchability at scale $s$ as

$q(x^{\mathcal{A}},x^{\mathcal{B}};s)=\mathcal{N}\big{(}0,s^{2}\mathbf{I}\big{)}\ast
p(x^{\mathcal{A}},x^{\mathcal{B}};0).$ (10)

Here $p(x^{\mathcal{A}},x^{\mathcal{B}};0)$ corresponds to the exact mapping
at infinite resolution. This can be interpreted as a diffusion in the
localization of the matches over scales. When multiple objects in a scene are
projected into images, so-called motion boundaries arise. These are
discontinuities in the matches which we illustrate in Figure 3. The diffusion
near these motion boundaries causes the conditional distribution to become
multimodal, explaining the need for multimodality in the coarse global
matching. Given an initial choice of $(x^{\mathcal{A}},x^{\mathcal{B}})$, as
in the refinement, the conditional distribution is unimodal locally. However,
if this initial choice is far outside the support of the distribution, using a
non-robust loss function is problematic. It is therefore motivated to use a
robust regression loss for this stage.

Loss formulation: Motivated by intuition and the theoretical model we now
propose our loss formulation from a probabilistic perspective, aiming to
minimize the Kullback–Leibler divergence between the estimated match
distribution at each scale, and the theoretical model distribution at that
scale. We begin by formulating the coarse loss. With non-overlapping bins as
defined in Section 3.3 the Kullback–Leibler divergence (where terms that are
constant w.r.t. $\theta$ are ignored) is

$\displaystyle D_{\rm
KL}(q(x^{\mathcal{B}},x^{\mathcal{A}};s)||p_{\text{coarse},\theta}(x^{\mathcal{B}},x^{\mathcal{A}}))=$
(11) $\displaystyle\mathbb{E}_{x^{\mathcal{A}},x^{\mathcal{B}}\sim
q}\big{[}-\log
p_{\text{coarse},\theta}(x^{\mathcal{B}}|x^{\mathcal{A}})p_{\text{coarse},\theta}(x^{\mathcal{A}})\big{]}=$
(12)
$\displaystyle-\int_{x^{\mathcal{A}},x^{\mathcal{B}}}\log\pi_{k^{{{\dagger}}}}(x^{\mathcal{A}})+\log
p_{\text{coarse},\theta}(x^{\mathcal{A}})dq,$ (13)

for $k^{{\dagger}}(x)={\rm argmin}_{k}\lVert m_{k}-x\rVert$ the index of the
closest anchor to $x$. Following DKM [17] we add a hyperparameter $\lambda$
that controls the weighting of the marginal compared to that of the
conditional as

$-\int_{x^{\mathcal{A}},x^{\mathcal{B}}}\log\pi_{k^{{\dagger}}}(x^{\mathcal{A}})+\lambda\log
p_{\text{coarse},\theta}(x^{\mathcal{A}})dq.$ (14)

Figure 4: Comparison of loss gradients. We use the generalized Charbonnier [3]
loss for refinement, which locally matches L2 gradients, but globally decays
with $|x|^{-1/2}$ toward zero.

In practice, we approximate $q$ with a discrete set of known correspondences
$\\{x^{\mathcal{A}},x^{\mathcal{B}}\\}$. Furthermore, to be consistent with
previous works [52, 17] we use a binary cross-entropy loss on
$p_{\text{coarse},\theta}(x^{\mathcal{A}})$. We call this loss
$\mathcal{L}_{\text{coarse}}$. We next discuss the fine loss
$\mathcal{L}_{\text{fine}}$.

We model the output of the refinement at scale $i$ as a generalized
Charbonnier [3] (with $\alpha=0.5$) distribution, for which the refiners
estimate the mean $\mu$. The generalized Charbonnier distribution behaves
locally like a Normal distribution, but has a flatter tail. When used as a
loss, the gradients behave locally like L2, but decay towards 0, see Figure 4.
Its logarithm, (ignoring terms that do not contribute to the gradient, and up-
to-scale) reads

$\displaystyle\log
p_{\theta}(x_{i}^{\mathcal{B}}|x_{i}^{\mathcal{A}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})=$
(15)
$\displaystyle-(||\mu_{\theta}(x_{i}^{\mathcal{A}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})-x_{i}^{\mathcal{B}}||^{2}+s)^{1/4},$
(16)

where
$\mu_{\theta}(x_{i}^{\mathcal{A}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})$
is the estimated mean of the distribution, and $s=2^{i}c$. In practice, we
choose $c=0.03$. The Kullback–Leibler divergence for each fine scale
$i\in\\{0,1,2,3\\}$ (where terms that are constant with respect to $\theta$
are ignored) reads

$\displaystyle D_{\rm
KL}(q(x_{i}^{\mathcal{B}},x_{i}^{\mathcal{A}};s=2^{i}c)||p_{i,\theta}(x_{i}^{\mathcal{B}},x_{i}^{\mathcal{A}}|\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1}))=$
(17) $\displaystyle\mathbb{E}_{x_{i}^{\mathcal{A}},x_{i}^{\mathcal{B}}\sim
q}\big{[}-(||\mu_{\theta}(x_{i}^{\mathcal{A}},\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})-x_{i}^{\mathcal{B}}||^{2}+s)^{1/4}\big{]}+$
$\displaystyle\mathbb{E}_{x_{i}^{\mathcal{A}},x_{i}^{\mathcal{B}}\sim
q}\big{[}-\log
p_{i,\theta}(x_{i}^{\mathcal{A}}|\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})\big{]}.$
(18)

In practice, we approximate $q$ with a discrete set of known correspondences
$\\{x^{\mathcal{A}},x^{\mathcal{B}}\\}$, and use a binary cross-entropy loss
on
$p_{\text{coarse},\theta}(x_{i}^{\mathcal{A}}|\hat{W}^{\mathcal{A}\to\mathcal{B}}_{i+1})$.
We sum over all fine scales to get the loss $\mathcal{L}_{\text{fine}}$.

Our combined loss yields:

$\mathcal{L}=\mathcal{L}_{\text{coarse}}+\mathcal{L}_{\text{fine}}.$ (19)

Note that we do not need to tune any scaling between these losses as the
coarse matching and fine stages are decoupled as gradients are cut in the
matching, and encoders are not shared.

Table 2: Ablation study. We systematically investigate the impact of our contributions, see Section 4.1 for detailed analysis. Measured in 100-percentage correct keypoints (PCK) (lower is better). Setup $\downarrow$ 100-PCK$@$ $\rightarrow$ | 1px | 3px | 5px
---|---|---|---
I (Baseline): DKM [17] | 17.0 | 7.3 | 5.8
II: I, $F_{\text{coarse},\theta}=\text{RN50}$, $F_{\text{fine},\theta}=\text{RN50}$ | 16.0 | 6.1 | 4.5
III: II, $F_{\text{fine},\theta}=\text{VGG19}$ | 14.5 | 5.4 | 4.5
IV: III, $D_{\theta}=\text{Transformer}$ | 14.4 | 5.4 | 4.1
V: IV, $F_{\text{coarse},\theta}=\text{DINOv2}$ | 14.3 | 4.6 | 3.2
VI: V, $\mathcal{L}_{\text{coarse}}=\text{reg.-by-class.}$ | 13.6 | 4.1 | 2.8
VII (Ours): VI, $\mathcal{L}_{\text{refine}}=\text{robust}$ | 13.1 | 4.0 | 2.7
VIII: VII, $D_{\theta}=\text{ConvNet}$ | 14.0 | 4.9 | 3.5

##  4 Experiments

###  4.1 Ablation Study

Here we investigate the impact of our contributions. We conduct all our
ablations on a validation test that we create. The validation set is made from
random pairs from the MegaDepth scenes $[0015,0022]$ with overlap $>0$. To
measure the performance we measure the percentage of estimated matches that
have an end-point-error (EPE) under a certain pixel threshold over all ground-
truth correspondences, which we call percent correct keypoints (PCK) using the
notation of previous work [52, 17].

Setup I consists of the same components as in DKM [17], retrained by us. In
Setup II we do not share weights between the fine and coarse features, which
improves performance due to specialization of the features. In Setup III we
replace the RN50 fine features with a VGG19, which further improves
performance. This is intriguing, as VGG19 features are worse performing when
used as coarse features as we show in Table 1. We then add the proposed
Transformer match decoder in Setup IV, however using the baseline regression
approach. Further, we incorporate the DINOv2 coarse features in Setup V, this
gives a significant improvement, owing to their significant robustness. Next,
in Setup VI change the loss function and output representation of the
Transformer match decoder $D_{\theta}$ to regression-by-classification, and
next in Setup VII use the robust regression loss. Both these changes further
significantly improve performance. This setup constitutes RoMa. When we change
back to the original ConvNet match decoder in Setup VIII from this final
setup, we find that the performance significantly drops, showing the
importance of the proposed Transformer match decoder.

Table 3: SotA comparison on IMC2022 [25]. Measured in mAA (higher is better). Method $\downarrow$ mAA $\rightarrow$ | $@10~{}\uparrow$
---|---
SiLK [21] | 68.6
SP [14]+SuperGlue [41] | 72.4
LoFTR [44] CVPR’21 | 78.3
MatchFormer [55] ACCV’22 | 78.3
QuadTree [46] ICLR’22 | 81.7
ASpanFormer [12] ECCV’22 | 83.8
DKM [17] CVPR’23 | 83.1
RoMa | 88.0
Table 4: SotA comparison on WxBS [35]. Measured in mAA at 10px (higher is better).  Method mAA$@$ $\rightarrow$ | $10\text{px}\uparrow$
---|---
DISK [53] NeurIps’20 | 35.5
DISK + LightGlue [53, 31] ICCV’23 | 41.7
SuperPoint +SuperGlue [14, 41] CVPR’20 | 31.4
LoFTR [44] CVPR’21 | 55.4
DKM [17] CVPR’23 | 58.9
RoMa | 80.1
Table 5: SotA comparison on MegaDepth-1500 [28, 44]. Measured in AUC (higher is better). Method $\downarrow$ AUC$@$ $\rightarrow$ |  $5^{\circ}$ $\uparrow$ |  $10^{\circ}$ $\uparrow$ |  $20^{\circ}$ $\uparrow$
---|---|---|---
LightGlue [31] ICCV’23 | 51.0 | 68.1 | 80.7
LoFTR [44] CVPR’21 | 52.8 | 69.2 | 81.2
PDC-Net+ [52] TPAMI’23 | 51.5 | 67.2 | 78.5
ASpanFormer [12] ECCV’22 | 55.3 | 71.5 | 83.1
ASTR [61] CVPR’23 | 58.4 | 73.1 | 83.8
DKM [17] CVPR’23 | 60.4 | 74.9 | 85.1
PMatch [63] CVPR’23 | 61.4 | 75.7 | 85.7
CasMTR [10] ICCV’23 | 59.1 | 74.3 | 84.8
RoMa | 62.6 | 76.7 | 86.3
Table 6: SotA comparison on ScanNet-1500 [13, 41]. Measured in AUC (higher is better). Method $\downarrow$ AUC$@$ $\rightarrow$ |  $5^{\circ}$ $\uparrow$ |  $10^{\circ}$ $\uparrow$ |  $20^{\circ}$ $\uparrow$
---|---|---|---
SuperGlue [41] CVPR’19 | 16.2 | 33.8 | 51.8
LoFTR [44] CVPR’21 | 22.1 | 40.8 | 57.6
PDC-Net+ [52] TPAMI’23 | 20.3 | 39.4 | 57.1
ASpanFormer [12] ECCV’22 | 25.6 | 46.0 | 63.3
PATS [36] CVPR’23 | 26.0 | 46.9 | 64.3
DKM [17] CVPR’23 | 29.4 | 50.7 | 68.3
PMatch [63] CVPR’23 | 29.4 | 50.1 | 67.4
CasMTR [10] ICCV’23 | 27.1 | 47.0 | 64.4
RoMa | 31.8 | 53.4 | 70.9

###  4.2 Training Setup

We use the training setup as in DKM [17]. Following DKM, we use a canonical
learning rate (for batchsize = 8) of $10^{-4}$ for the decoder, and $5\cdot
10^{-6}$ for the encoder(s). We use the same training split as in DKM, which
consists of randomly sampled pairs from the MegaDepth and ScanNet sets
excluding the scenes used for testing. The supervised warps are derived from
dense depth maps from multi-view-stereo (MVS) of SfM reconstructions in the
case of MegaDepth, and from RGB-D for ScanNet. Following previous work [44,
12, 17], use a model trained on the ScanNet training set when evaluating on
ScanNet-1500. All other evaluation is done on a model trained only on
MegaDepth.

As in DKM we train both the coarse matching and refinement networks jointly.
Note that since we detach gradients between the coarse matching and
refinement, the network could in principle also be trained in two stages. For
results used in the ablation, we used a resolution of $448\times 448$, and for
the final method we trained on a resolution of $560\times 560$.

###  4.3 Two-View Geometry

We evaluate on a diverse set of two-view geometry benchmarks. We follow DKM
[17] and sample correspondences using a balanced sampling approach, producing
$10000$ matches, which are then used for estimation. We consistently improve
compared to prior work across the board, in particular achieving a relative
error reduction on the competitive IMC2022 [25] benchmark by 26%, and a gain
of 36% in performance on the exceptionally difficult WxBS [35] benchmark.

Image Matching Challenge 2022: We submit to the 2022 version of the image
matching challenge [25], which consists of a hidden test-set of Google street-
view images with the task to estimate the fundamental matrix between them. We
present results in Table 3. RoMa attains significant improvements compared to
previous approaches, with a relative error reduction of 26% compared to the
previous best approach.

WxBS Benchmark: We evaluate RoMa on the extremely difficult WxBS benchmark
[35], version 1.1 with updated ground truth and evaluation
protocol444https://ducha-aiki.github.io/wide-baseline-stereo-
blog/2021/07/30/Reviving-WxBS-benchmark. The metric is mean average precision
on ground truth correspondences consistent with the estimated fundamental
matrix at a 10 pixel threshold. All methods use MAGSAC++ [2] as implemented in
OpenCV. Results are presented in Table 4. Here we achieve an outstanding
improvement of 36% compared to the state-of-the-art. We attribute these major
gains to the superior robustness of RoMa compared to previous approaches. We
qualitatively present examples of this in the supplementary.

MegaDepth-1500 Pose Estimation: We use the MegaDepth-1500 test set [44] which
consists of 1500 pairs from scene 0015 (St. Peter’s Basilica) and 0022
(Brandenburger Tor). We follow the protocol in [44, 12] and use a RANSAC
threshold of 0.5. Results are presented in Table 5.

ScanNet-1500 Pose Estimation: ScanNet [13] is a large scale indoor dataset,
composed of challenging sequences with low texture regions and large changes
in perspective. We follow the evaluation in SuperGlue [41]. Results are
presented in Table 6. We achieve state-of-the-art results, achieving the first
AUC$@$20∘ scores over 70.

MegaDepth-8-Scenes: We evaluate RoMa on the Megadepth-8-Scenes benchmark [28,
17]. We present results in Table 7. Here too we outperform previous
approaches.

Table 7:  SotA comparison on Megadepth-8-Scenes [17]. Measured in AUC (higher is better). Method $\downarrow$ AUC $\rightarrow$ | $@5^{\circ}$ | $@10^{\circ}$ | $@20^{\circ}$
---|---|---|---
PDCNet+ [52] TPAMI’23 | 51.8 | 66.6 | 77.2
ASpanFormer [12] ECCV’22 | 57.2 | 72.1 | 82.9
DKM [17] CVPR’23 | 60.5 | 74.5 | 84.2
RoMa | 62.2 | 75.9 | 85.3

###  4.4 Visual Localization

We evaluate RoMa on the InLoc [45] Visual Localization benchmark, using the
HLoc [40] pipeline. We follow the approach in DKM [17] to sample
correspondences. Results are presented in Table 8. We show large improvements
compared to all previous approaches, setting a new state-of-the-art.

Table 8: SotA comparison on InLoc [45]. We report the percentage of query images localized within 0.25/0.5/1.0 meters and 2/5/10∘ of the ground-truth pose (higher is better). Method | DUC1 | DUC2
---|---|---
|  (0.25m,2∘)/(0.5m,5∘)/(1.0m,10∘)
PATS | 55.6 / 71.2 / 81.0 | 58.8 / 80.9 / 85.5
DKM | 51.5 / 75.3 / 86.9 |  63.4 / 82.4 / 87.8
CasMTR | 53.5 / 76.8 / 85.4 | 51.9 / 70.2 / 83.2
RoMa |  60.6 / 79.3 / 89.9 |  66.4 / 83.2 / 87.8

###  4.5 Runtime Comparison

We compare the runtime of RoMa and the baseline DKM at a resolution of
$560\times 560$ at a batch size of 8 on an RTX6000 GPU. We observe a modest
7$\%$ increase in runtime from 186.3 $\to$ 198.8 ms per pair.

##  5 Conclusion

We have presented RoMa, a robust dense feature matcher. Our model leverages
frozen pretrained coarse features from the foundation model DINOv2 together
with specialized ConvNet fine features, creating a precisely localizable and
robust feature pyramid. We further improved performance with our proposed
tailored transformer match decoder, which predicts anchor probabilities
instead of regressing coordinates. Finally, we proposed an improved loss
formulation through regression-by-classification with subsequent robust
regression. Our comprehensive experiments show that RoMa achieves major gains
across the board, setting a new state-of-the-art. In particular, our biggest
gains (36% increase on WxBS [35]) are achieved on the most difficult
benchmarks, highlighting the robustness of our approach. Code is provided at
github.com/Parskatt/RoMa.

Limitations and Future Work:

  1. (a)

Our approach relies on supervised correspondences, which limits the amount of
usable data. We remedied this by using pretrained frozen foundation model
features, which improves generalization.

  2. (b)

We train on the task of dense feature matching which is an indirect way of
optimizing for the downstream tasks of two-view geometry, localization, or 3D
reconstruction. Directly training on the downstream tasks could improve
performance.

## References

  * Balntas et al. [2017] Vassileios Balntas, Karel Lenc, Andrea Vedaldi, and Krystian Mikolajczyk.  HPatches: A benchmark and evaluation of handcrafted and learned local descriptors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5173–5182, 2017.
  * Barath et al. [2020] Daniel Barath, Jana Noskova, Maksym Ivashechkin, and Jiri Matas.  MAGSAC++, a fast, reliable and accurate robust estimator.  In _Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Barron [2019] Jonathan T Barron.  A general and adaptive robust loss function.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4331–4339, 2019.
  * Bay et al. [2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In _European conference on computer vision_ , pages 404–417. Springer, 2006.
  * Black and Anandan [1996] Michael J Black and Paul Anandan.  The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields.  _Computer vision and image understanding_ , 63(1):75–104, 1996.
  * Black and Rangarajan [1996] Michael J Black and Anand Rangarajan.  On the unification of line processes, outlier rejection, and robust statistics with applications in early vision.  _International journal of computer vision_ , 19(1):57–91, 1996.
  * Bökman and Kahl [2022] Georg Bökman and Fredrik Kahl.  A case for using rotation invariant features in state of the art feature matchers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5110–5119, 2022.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Budvytis et al. [2019] Ignas Budvytis, Marvin Teichmann, Tomas Vojir, and Roberto Cipolla.  Large scale joint semantic re-localisation and scene understanding via globally unique instance coordinate regression.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , pages 86.1–86.13. BMVA Press, 2019.
  * Cao and Fu [2023] Chenjie Cao and Yanwei Fu.  Improving transformer-based image matching by cascaded capturing spatially informative keypoints.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 12129–12139, 2023.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chen et al. [2022] Hongkai Chen, Zixin Luo, Lei Zhou, Yurun Tian, Mingmin Zhen, Tian Fang, David Mckinnon, Yanghai Tsin, and Long Quan.  ASpanFormer: Detector-free image matching with adaptive span transformer.  In _Proc. European Conference on Computer Vision (ECCV)_ , 2022.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5828–5839, 2017.
  * DeTone et al. [2018] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 224–236, 2018.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
  * Dusmanu et al. [2019] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  In _Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019.
  * Edstedt et al. [2023] Johan Edstedt, Ioannis Athanasiadis, Mårten Wadenbäck, and Michael Felsberg.  DKM: Dense kernelized feature matching for geometry estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Felsberg et al. [2006] Michael Felsberg, P-E Forssen, and H Scharr.  Channel smoothing: Efficient robust smoothing of low-level signal features.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 28(2):209–222, 2006.
  * Garg et al. [2020] Divyansh Garg, Yan Wang, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao.  Wasserstein distances for stereo disparity estimation.  _Advances in Neural Information Processing Systems_ , 33:22517–22529, 2020.
  * Germain et al. [2021] Hugo Germain, Vincent Lepetit, and Guillaume Bourmaud.  Neural reprojection error: Merging feature learning and camera pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 414–423, 2021.
  * Gleize et al. [2023] Pierre Gleize, Weiyao Wang, and Matt Feiszli.  SiLK: Simple Learned Keypoints.  In _ICCV_ , 2023.
  * Häger et al. [2021] Gustav Häger, Mikael Persson, and Michael Felsberg.  Predicting disparity distributions.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4363–4369. IEEE, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16000–16009, 2022.
  * Howard et al. [2022] Addison Howard, Eduard Trulls, Kwang Moo Yi, Dmitry Mishkin, Sohier Dane, and Yuhe Jin.  Image matching challenge 2022, 2022.
  * Koenderink [1984] Jan J Koenderink.  The structure of images.  _Biological cybernetics_ , 50(5):363–370, 1984.
  * Li et al. [2020] Xiaotian Li, Shuzhe Wang, Yi Zhao, Jakob Verbeek, and Juho Kannala.  Hierarchical scene coordinate classification and regression for visual localization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11983–11992, 2020.
  * Li and Snavely [2018] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2041–2050, 2018.
  * Lin et al. [2022] Yutong Lin, Ze Liu, Zheng Zhang, Han Hu, Nanning Zheng, Stephen Lin, and Yue Cao.  Could giant pre-trained image models extract universal representations?  _Advances in Neural Information Processing Systems_ , 35:8332–8346, 2022.
  * Lindeberg [1994] Tony Lindeberg.  Scale-space theory: A basic tool for analyzing structures at different scales.  _Journal of applied statistics_ , 21(1-2):225–270, 1994.
  * Lindenberger et al. [2023] Philipp Lindenberger, Paul-Edouard Sarlin, and Marc Pollefeys.  LightGlue: Local Feature Matching at Light Speed.  In _ICCV_ , 2023.
  * Liu et al. [2022] Haisong Liu, Tao Lu, Yihui Xu, Jia Liu, Wenjie Li, and Lijun Chen.  Camliflow: bidirectional camera-lidar fusion for joint optical flow and scene flow estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5791–5801, 2022.
  * Lowe [2004] David G Lowe.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ , 60(2):91–110, 2004.
  * Melekhov et al. [2019] Iaroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Marc Pollefeys, Esa Rahtu, and Juho Kannala.  Dgc-net: Dense geometric correspondence network.  In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 1034–1042. IEEE, 2019.
  * Mishkin et al. [2015] Dmytro Mishkin, Jiri Matas, Michal Perdoch, and Karel Lenc.  WxBS: Wide Baseline Stereo Generalizations.  In _Proceedings of the British Machine Vision Conference_. BMVA, 2015.
  * Ni et al. [2023] Junjie Ni, Yijin Li, Zhaoyang Huang, Hongsheng Li, Hujun Bao, Zhaopeng Cui, and Guofeng Zhang.  Pats: Patch area transportation with subdivision for local feature matching.  In _The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR)_ , 2023.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  DINOv2: Learning robust visual features without supervision.  _arXiv:2304.07193_ , 2023.
  * Rasmussen and Williams [2005] Carl Edward Rasmussen and Christopher K. I. Williams.  _Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning)_.  The MIT Press, 2005.
  * Revaud et al. [2019] Jerome Revaud, Cesar De Souza, Martin Humenberger, and Philippe Weinzaepfel.  R2d2: Reliable and repeatable detector and descriptor.  _Advances in neural information processing systems_ , 32:12405–12415, 2019.
  * Sarlin et al. [2019] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk.  From coarse to fine: Robust hierarchical localization at large scale.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12716–12725, 2019.
  * Sarlin et al. [2020] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superglue: Learning feature matching with graph neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4938–4947, 2020.
  * Sarlin et al. [2021] Paul-Edouard Sarlin, Ajaykumar Unagar, Mans Larsson, Hugo Germain, Carl Toft, Viktor Larsson, Marc Pollefeys, Vincent Lepetit, Lars Hammarstrand, Fredrik Kahl, et al.  Back to the feature: Learning robust camera localization from pixels to pose.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3247–3257, 2021.
  * Schonberger and Frahm [2016] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4104–4113, 2016.
  * Sun et al. [2021] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8922–8931, 2021.
  * Taira et al. [2018] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7199–7209, 2018.
  * Tang et al. [2022] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan.  Quadtree attention for vision transformers.  In _International Conference on Learning Representations_ , 2022.
  * Tian et al. [2020] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia.  Prior guided feature enrichment network for few-shot segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(2):1050–1065, 2020.
  * Torgo and Gama [1996] Luís Torgo and João Gama.  Regression by classification.  In _Advances in Artificial Intelligence_ , pages 51–60, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg.
  * Truong et al. [2020a] Prune Truong, Martin Danelljan, Luc V Gool, and Radu Timofte.  GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network.  _Advances in Neural Information Processing Systems_ , 33, 2020a.
  * Truong et al. [2020b] Prune Truong, Martin Danelljan, and Radu Timofte.  GLU-Net: Global-local universal network for dense flow and correspondences.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6258–6268, 2020b.
  * Truong et al. [2021] Prune Truong, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning accurate dense correspondences and when to trust them.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5714–5724, 2021.
  * Truong et al. [2023] Prune Truong, Martin Danelljan, Radu Timofte, and Luc Van Gool.  PDC-Net+: Enhanced Probabilistic Dense Correspondence Network.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Tyszkiewicz et al. [2020] Michal J. Tyszkiewicz, Pascal Fua, and Eduard Trulls.  DISK: learning local features with policy gradient.  In _NeurIPS_ , 2020.
  * Vasconcelos et al. [2022] Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin.  Proper reuse of image classification features improves object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 13628–13637, 2022.
  * Wang et al. [2022] Qing Wang, Jiaming Zhang, Kailun Yang, Kunyu Peng, and Rainer Stiefelhagen.  MatchFormer: Interleaving attention in transformers for feature matching.  In _Asian Conference on Computer Vision_ , 2022.
  * Wei et al. [2022] Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer.  Masked feature prediction for self-supervised visual pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14668–14678, 2022.
  * Weiss and Indurkhya [1993] Sholom M. Weiss and Nitin Indurkhya.  Rule-based regression.  In _Proceedings of the 13th International Joint Conference on Artificial Intelligence. Chambéry, France, August 28 - September 3, 1993_ , pages 1072–1078. Morgan Kaufmann, 1993.
  * Weiss and Indurkhya [1995] Sholom M. Weiss and Nitin Indurkhya.  Rule-based machine learning methods for functional prediction.  _J. Artif. Intell. Res._ , 3:383–403, 1995.
  * Witkin [1983] Andrew P. Witkin.  Scale space filtering.  _Proc. 8th International Joint on Artificial Intelligence_ , pages 1091–1022, 1983.
  * Xie et al. [2023] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao.  Revealing the dark secrets of masked image modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14475–14485, 2023.
  * Yu et al. [2023] Jiahuan Yu, Jiahao Chang, Jianfeng He, Tianzhu Zhang, Jiyang Yu, and Wu Feng.  ASTR: Adaptive spot-guided transformer for consistent local feature matching.  In _The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR)_ , 2023.
  * Zhou et al. [2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer.  In _International Conference on Learning Representations_ , 2022.
  * Zhu and Liu [2023] Shengjie Zhu and Xiaoming Liu.  PMatch: Paired masked image modeling for dense geometric matching.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.

Supplementary Material

Figure 5: Evaluation of frozen features. From top to bottom: Image pair, VGG19
matches, RN50 matches, DINOv2 matches, RoMa matches. DINOv2 is significantly
more robust than the VGG19 and RN50. Quantitative results are presented in
Table 1.

In this supplementary material, we provide further details and qualitative
examples that could not fit into the main text of the paper.

##  Appendix A Further Details on Frozen Feature Evaluation

We use an exponential cosine kernel as in DKM [17] with an inverse temperature
of 10. We train using the same training split as in our main experiments,
using the same learning rates (note that we only train a single linear layer,
as the backbone is frozen). We use the regression-by-classification loss that
we proposed in Section 3.4. We present a qualitative example of the estimated
warps from the frozen features in Figure 5.

##  Appendix B Further Architectural Details

Encoders: We extract fine features of stride $\\{1,2,4,8\\}$ by taking the
outputs of the layer before each $2\times 2$ maxpool. These have dimension
$\\{64,128,256,512\\}$ respectively. We project these with a linear layer
followed by batchnorm to dimension $\\{9,64,256,512\\}$.

We use the patch features from DINOv2 [37] and do not use the cls token. We
use the ViT-L-14 model, with patch size 14 and dimension $1024$. We linearly
project these features (with batchnorm) to dimension $512$.

Global Matcher: We use a Gaussian Process [38] match encoder as in DKM [17].
We use an exponential cosine kernel [17], with inverse temperature 10. As in
DKM, the GP predicts a posterior over embedded coordinates in the other image.
We use an embedding space of dimension $512$.

For details on $D_{\theta}$ we refer to Section 3.3.

Refiners: Following Edstedt et al. [17] we use 5 refiners at strides
$\\{1,2,4,8,14\\}$. They each consist of 8 convolutional blocks. The internal
dimension is set to $\\{24,144,569,1137,1377\\}$. The input to the refiners
are the stacked feature maps, local correlation around the previous warp of
size $\\{0,0,5,7,15\\}$, as well as a linear encoding of the previous warp.
The output is a $B\times H_{s}\times W_{s}\times(2+1)$ tensor, containing the
warp and an logit offset to the certainty.

##  Appendix C Qualitative Comparison on WxBS

We qualitatively compare estimated matches from RoMa and DKM on the WxBS
benchmark in Figure 6. DKM fails on multiple pairs on this dataset, while RoMa
is more robust. In particular, RoMa is able to match even for changes is
season (bottom right), extreme illumination (bottom left, top left), and
extreme scale and viewpoint (top right).

##  Appendix D Further Details on Metrics

Image Matching Challenge 2022: The mean average accuracy (mAA) metric is
computed between the estimated fundamental matrix and the hidden ground truth.
The error in terms of rotation in degrees and translation in meters. Given one
threshold over each, a pose is classified as accurate if it meets both
thresholds. This is done over ten pairs of uniformly spaced thresholds. The
mAA is then the average over the threshold and over the images (balanced
across the scenes).

MegaDepth/ScanNet: The AUC metric used measures the error of the estimated
Essential matrix compared to the ground truth. The error per pair is the
maximum of the rotational and translational error. As there is no metric scale
available, the translational error is measured in the cosine angle. The recall
at a threshold $\tau$ is the percentage of pairs with an error lower than
$\tau$. The AUC$@\tau^{\circ}$ is the integral over the recall as a function
of the thresholds, up to $\tau$, divided by $\tau$. In practice, this is
approximated by the trapezial rule over all errors of the method over the
dataset.

##  Appendix E Further Details on Theoretical Model

Here we discuss a simple connection to scale-space theory, that did not fit in
the main paper. Our theoretical model of matchability in Section 3.4 has a
straightforward connection to scale-space theory [59, 26, 30]. The image
scale-space is parameterized by a parameter $s$,

$L(x,s)=\int g(x-y;s)I(y)dy,$ (20)

where

$g(x;s)=\frac{1}{2\pi s^{2}}\exp\bigg{(}-\frac{1}{2s^{2}}\lVert
x\rVert^{2}\bigg{)}$ (21)

is a Gaussian kernel. Applying this kernel jointly on the matching
distribution yields the diffusion process in the paper.

##  Appendix F Further Details on Match Sampling

Dense feature matching methods produce a dense warp and certainty. However,
most robust relative pose estimators (used in the downstream two-view pose
estimation evaluation) assume a sparse set of correspondences. While one could
in principle use all correspondences from the warp, this is prohibitively
expensive in practice. We instead follow the approach of DKM [17] and use a
balanced sampling approach to produce a sparse set of matches. The balanced
sampling approach uses a KDE estimate of the match distribution
$p_{\theta}(x^{\mathcal{A}},x^{\mathcal{B}})$ to rebalance the distribution of
the samples, by reweighting the samples with the reciprocal of the KDE. This
increases the number of matches in less certain regions, which Edstedt et al.
[17] demonstrated improves performance.

Figure 6: Qualitative comparison. RoMa is significantly more robust to extreme
changes in viewpoint and illumination than DKM.


# Exploratory Analysis of Academic Collaborations between French and US

George Panagopoulos, Michalis Vazirgiannis
<EMAIL_ADDRESS>
LIX, Ecole Polytechnique
Office for Science and Technology of the Embassy of France in the US

(January 2021)

##  1 Introduction

International academic collaborations cultivate diversity in the research
landscape and facilitate multiperspective methods, as the scope of each
country’s science depends on its needs, history, wealth etc. Moreover the
quality of science differ significantly amongst nations[5], which renders
international collaborations a potential source to understand the dynamics
between countries and their advancements. Analyzing these collaborations can
reveal sharing expertise between two countries in different fields, the most
well-known institutions of a nation, the overall success of collaborative
efforts compared to local ones etc. Such analysis were initially performed
using statistical metrics [7], but network analysis has later proven much more
expressive [14, 4]. In this exploratory analysis, we aim to examine the
collaboration patterns between French and US institutions. Towards this, we
capitalize on the Microsoft Academic Graph MAG [13], the largest open
bibliographic dataset that contains detailed information for authors,
publications and institutions. We use the coordinates of the world map to
tally affiliations to France or USA. In cases where the coordinates of an
affiliation were absent, we used its Wikipedia url and named entity
recognition to identify the country of its address in the Wikipedia page. We
need to stress that institute names have been volatile (due to University
federations created) in the last decade in France, so this is a best effort
trial. The results indicate an intensive and increasing scientific production
in with , with certain institutions such as Harvard, MIT and CNRS standing
out.

##  2 Analysis

###  2.1 Coauthorships of the top French Institutes

We define a collaboration among a French and a US scientific institute if
there is at least one paper coauthored by scientists from both these
institutes. Among the French academic institutes that collaborate with USA we
report in Table 1 the 10 most productive (in terms of number of papers) and
their most frequent collaborators in the USA. Figure 1 visualizes the same
information with a chord plot, where the edges are colored based on the US
institutes. This shows the US universities collaborating mostly with the
aforementioned most productive French institutes. Harvard and MIT stand out
with more than 3 collaborations with one of the top French institutes, while
the strongest collaboration takes place between CNRS and CalTEch.

Figure 1: Table 1 visualized in a chord plot France | USA
---|---
Centre national de la recherche scientifique (CNRS) |  | California Institute of Technology (CalTech) [2678]
---
Massachusetts Institute of Technology(MIT) [2226]
Harvard University [2106]
French Institute of Health and Medical Research (FIHM) |  | Harvard University [1765]
---
National Institutes of Health (NIH) [1484]
Boston Children’s Hospital (BCH) [1053]
University of Paris (UParis) |  | Harvard University [2061]
---
Stanford University [1486]
Massachusetts Institute of Technology (MIT) [1280]
École Normale Supérieure (ENS) |  | Harvard University [276]
---
Massachusetts Institute of Technology (MIT) [252]
University of California, Berkeley [247]
Institut National de la Recherche Agronomique (INRA) |  | University of California, Davis [255]
---
Agricultural Research Service (AGS) [255]
United States Department of Agriculture (USDA) [244]
Pierre-and-Marie-Curie University (UPMC) |  | Massachusetts Institute of Technology (MIT) [769]
---
Harvard University [761]
University of Michigan [670]
University of Bordeaux |  | Harvard University [335]
---
University of Washington [277]
National Institutes of Health (NIH) [212]
University of Paris-Sud |  | Massachusetts Institute of Technology (MIT) [1327]
---
Harvard University [1305]
Iowa State University [1093]
University of Montpellier |  | University of Washington [230]
---
Harvard University [219]
University of Maryland, College Park (UMD) [211]
Pasteur Institute |  | National Institutes of Health (NIH) [561]
---
Harvard University [421]
Centers for Disease Control and Prevention (CDCP) [313]
Table 1: Top 10 French institutions (in terms of numbers of papers), with
their 3 closest collaborators in US. The number of collaborations in each case
is included after the name of the US institute. The names of the institutes
are abbreviated for visualization purposes on Fig. 1.

###  2.2 The timeline of French-US collaborations

The oldest collaboration between a French and a US institute dates back to
1930, with a joint paper between Ecole Normal Superiere and Cornell University
entitled ”LA GRAISSE DU SANG ET LA GRAISSE DU LAIT PENDANT LA. LACTATION” [12]
. Since then, the number of collaborations, as well as their impact (in terms
of citations) have increased significantly, as one can see in Figures 2 and 3.
There is an almost exponential increase in the number of papers produced by
joint collaborations. The same applies for the impact of these works, which
increases especially until 2010.Naturally, the citations count in recent years
is diminished, as young papers have fewer citations and they need time to get
cited. There are also some monumental years, such as 2012, where the inclusion
of the publications on the discovery of the Higgs Boson particle e.g. [1, 2]
has produced a massive number of citations.

Figure 2: FR-US joint papers temporal distribution. Figure 3: FR-US joint
papers’ citations temporal distribution .

Distinguishing between different fields, we can see in Fig. 4 how the number
of collaborations have evolved through the years in different disciplines, as
well as their success in Fig. 5. Bare in mind that a paper might belong to
more than one fields.

  * •

We see that the majority of works are comprised of medical studies, which is
quite common in academia.

  * •

The second field is computer science and the third biology. Especially in
computer science, there is a steep increase around 2010.

  * •

The citations do not follow the same pattern, as the top cited domain is
medicine, followed by biology and and computer science. This is due to the
known differences in citation patterns among computer science and biology
[11].

  * •

Mathematics is the least active field in this context since publications in
this area are relatively rare. Still there are some spikes of citations
through the years because of some important papers.

Figure 4: FR-US joint papers temporal distribution per prominent scientific
area. Figure 5: FR-US joint papers citations temporal distribution per
prominent scientific area.

####  2.2.1 Top Collaborations

Let the strength of the collaboration $c$ be denoted as $P(c)$, refering to
the number of such papers. We rank all collaborations based on strength and
define the most prominant the ones who are above the average number of papers
in all collaborations and six standard deviations of the distribution i.e.
$\\{c|P(c)\geq mean({\\{P(c^{\prime})\\}_{c^{\prime}\in
C}})+6sd(\\{P(c^{\prime})\\}_{c^{\prime}\in C})\\}$, where $C$ is the set of
all collaborations. The threshold and the density are visualized in figure 6.
The number of collaborations that are above this threshold is 492.

Figure 6: Density of the number of collaborations with the number of papers,
and the threshold to define the top collaborations.

These are visualized in the map plot 7. Although we can get an idea of the
overall cities that collaborate mostly with each other, it is still a very
perplex image to make sense of. Thus we reduce it even more, by taking the top
100 collaborations, and making a weighted bipartite plot in 8.

Figure 7: USA-France top collaborations. Figure 8: USA-France top 100
collaborations in terms of number of joint papers. Left is USA, right is
France. The size of the nodes is proportional to their number of papers, the
edge width is proportional to the collaborations’ strength and their color is
different for each French institution.

Few observations derived from a first glance on this network:

  * •

CNRS is the French institute with the most collaborations, followed by Ecole
Polytechnique and University of Strasbourg.

  * •

Some strong collaborations that stand out are CNRS with CalTech and UMD and
University of Paris with Harvard.

  * •

The French Institute of Health and Medical Research has few collaborations,
but with two US institutes well known for their achievements in medicine,
Harvard and NIH.

  * •

Institutes performing research on fields like physics, biology or chemistry
tend to have more connections than the ones focusing on language studies or
the ones performing solely medical research. This might indicate that STEM
projects applied or related to medicine exhibit international collaborations,
while purely medical studies have a local collaboration pattern.

  * •

The most productive French Institute is clearly CNRS, followed by University
of Paris, while for the USA it is Harvard and University of Michigan.

####  2.2.2 Top 10 collaborations

The top 10 USA-France collaborations in terms of absolute number of joint
papers can be seen in table 2

USA | France
---|---
California Institute of Technology | Centre national de la recherche scientifique (CNRS) 2678
Massachusetts Institute of Technology | Centre national de la recherche scientifique (CNRS) 2226
Harvard University | Centre national de la recherche scientifique (CNRS) 2106
University of Maryland, College Park | Centre national de la recherche scientifique (CNRS) 2092
Harvard University | University of Paris 2061
Ohio State University | Centre national de la recherche scientifique (CNRS) 1840
Harvard University | French Institute of Health and Medical Research 1765
University of Wisconsin-Madison | Centre national de la recherche scientifique (CNRS) 1742
Princeton University | Centre national de la recherche scientifique (CNRS) 1638
University of Michigan | Centre national de la recherche scientifique (CNRS) 1596
Table 2: Top USA-France collaborations in terms of number of papers.

For each of the 10 collaborations we report the topical distribution in the
figures below. Some observations:

  * •

The collaboration between CalTech and CNRS focuses mainly in math,chemistry
and physics, since both are widely known for their excellence in these fields.

  * •

The joint works of CNRS with MIT and Harvard are predominantly related to
medicine. In contrast, the secondary fields in MIT are computer science,
chemistry, physics and math, while for Harvard biology is second and the rest
follow. This makes sense because, as mentioned above, MIT is focusing more on
science and technology while Harvard has a broader scope.

  * •

University of Maryland (UMD) and CNRS seem to collaborate in a variety of
disciplines, with physics being the dominant.

  * •

It is clear the main joint papers between Harvard and University of Paris
refer to medical studies. The same applied to the French Institute of Health
and Medical Research, which overall might refer to joint works between these
three institutes.

  * •

CNRS has also collaborated extensively with Ohio State, Wisconsin-Madison and
Michigan, especially in medicine.

##  3 Future Work

These initial results indicate there is an intensive and increasing scientific
production in terms of joint papers and resulting citations. Overall, several
new hypotheses can be tested:

  * •

The use of influence and success metrics to identify the most crucial authors
[3, 10, 8], cliques or laboratories that guide the course of the majority of
the collaborations in a direct or indirect manner.

  * •

Prediction of new collaborations based on the fields the institutes belong to
or the venues they publish at.

  * •

Prediction of institutes that will increase their long-term impact based on
their current position and activity in the network [9].

  * •

Use of complementary datasets such as OpenAIRE [6] to include information
about funded projects.

##  4 Acknowledgements

This analysis was performed after the request of Dr. Yves Frenot, Dr. Jean-
Baptiste Bordes and Maxime Benallaoua from the Office for Science and
Technology of the Embassy of France in the United-States, with whom we
collaborated to set the hypotheses examined.

## References

  * [1] Aad, G., Abajyan, T., Abbott, B., Abdallah, J., Khalek, S.A., Abdelalim, A.A., Abdinov, O., Aben, R., Abi, B., Abolins, M., et al.: Observation of a new particle in the search for the standard model higgs boson with the atlas detector at the lhc.  Physics Letters B 716(1), 1–29 (2012)
  * [2] Bezrukov, F., Kalmykov, M.Y., Kniehl, B.A., Shaposhnikov, M.: Higgs boson mass and new physics.  Journal of High Energy Physics 2012(10), 140 (2012)
  * [3] Giatsidis, C., Nikolentzos, G., Zhang, C., Tang, J., Vazirgiannis, M.: Rooted citation graphs density metrics for research papers influence evaluation.  Journal of Informetrics 13(2), 757–768 (2019)
  * [4] González-Alcaide, G., Aleixandre-Benavent, R., Navarro-Molina, C., Valderrama-Zurián, J.C.: Coauthorship networks and institutional collaboration patterns in reproductive biology.  Fertility and sterility 90(4), 941–956 (2008)
  * [5] King, D.A.: The scientific impact of nations.  Nature 430(6997), 311 (2004)
  * [6] Manghi, P., Manola, N., Horstmann, W., Peters, D.: An infrastructure for managing ec funded research output-the openaire project.  The Grey Journal (TGJ): An International Journal on Grey Literature 6(1) (2010)
  * [7] Melin, G., Persson, O.: Studying research collaboration using co-authorships.  Scientometrics 36(3), 363–377 (1996)
  * [8] Panagopoulos, G., Malliaros, F., Vazirgiannis, M.: Multi-task learning for influence estimation and maximization.  IEEE Transactions on Knowledge and Data Engineering (2020)
  * [9] Panagopoulos, G., Tsatsaronis, G., Varlamis, I.: Detecting rising stars in dynamic collaborative networks.  Journal of Informetrics 11(1), 198–222 (2017)
  * [10] Panagopoulos, G., Xypolopoulos, C., Skianis, K., Giatsidis, C., Tang, J., Vazirgiannis, M.: Scientometrics for success and influence in the microsoft academic graph.  In: International Conference on Complex Networks and Their Applications, pp. 1007–1017. Springer (2019)
  * [11] Patience, G.S., Patience, C.A., Blais, B., Bertrand, F.: Citation analysis of scientific categories.  Heliyon 3(5), e00,300 (2017)
  * [12] Porcher, C., Maynard, L.: La graisse du sang et la graisse du lait pendant la lactation.  Le Lait 10(96), 601–613 (1930)
  * [13] Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.j.P., Wang, K.: An overview of microsoft academic service (mas) and applications.  In: Proceedings of the 24th international conference on world wide web, pp. 243–246. ACM (2015)
  * [14] Wagner, C.S., Leydesdorff, L.: Mapping the network of global science: comparing international co-authorships from 1990 to 2000.  International journal of Technology and Globalisation 1(2), 185–208 (2005)