11institutetext: School of Automation, Southeast University, Nanjing 210096,
China
11email<EMAIL_ADDRESS>
# RADA: Robust and Accurate Feature Learning with Domain Adaptation

Jingtai He  Gehao Zhang  Tingting Liu  Songlin Du

###### Abstract

Recent advancements in keypoint detection and descriptor extraction have shown
impressive performance in local feature learning tasks. However, existing
methods generally exhibit suboptimal performance under extreme conditions such
as significant appearance changes and domain shifts. In this study, we
introduce a multi-level feature aggregation network that incorporates two
pivotal components to facilitate the learning of robust and accurate features
with domain adaptation. First, we employ domain adaptation supervision to
align high-level feature distributions across different domains to achieve
invariant domain representations. Second, we propose a Transformer-based
booster that enhances descriptor robustness by integrating visual and
geometric information through wave position encoding concepts, effectively
handling complex conditions. To ensure the accuracy and robustness of
features, we adopt a hierarchical architecture to capture comprehensive
information and apply meticulous targeted supervision to keypoint detection,
descriptor extraction, and their coupled processing. Extensive experiments
demonstrate that our method, RADA, achieves excellent results in image
matching, camera pose estimation, and visual localization tasks.

###### Keywords:

Local features Domain adaptation Descriptor enhancement Image matching

##  1 Introduction

Robust and accurate keypoint detection and descriptor extraction are
fundamental for various computer vision tasks, such as visual localization
[23], and structure from motion (SfM) [11]. With the advent of deep learning,
significant advancements [5, 6, 14] have been achieved, enhancing both the
accuracy and efficiency of local features compared to traditional handcrafted
methods [10, 17].

The adoption of sparse-to-sparse feature learning, which follows a ’detect-
then-describe’ paradigm, has proven common and effective in feature matching
methodologies [31]. Numerous methods adhering to this paradigm have been
developed, focusing on optimizing individual components, such as keypoint
detection or descriptor extraction, to enhance the local features.
Nevertheless, the robustness of these features can significantly deteriorate
under extreme domain variations, such as day-night changes or different
seasons. This deterioration occurs because the keypoint detection stage is
susceptible to variations in low-level image statistics, which affects the
subsequent feature descriptors’ ability to maintain invariance. To mitigate
this challenge, we draw inspiration from transfer learning [7] and propose
domain adaptation supervision implemented prior to the keypoint detection
stage. This component aims to establish a mapping between different domains,
thereby minimizing discrepancies in high-level feature distributions. By
aligning these distributions, we improve keypoint detector stability, the
robustness and invarance of descriptors across varying conditions.

Under large illumination and viewpoint changes, local visual information
becomes unreliable and indistinguishable, leading to the degradation of
descriptors. To address this issue, the global modeling capability of
Transformer has been increasingly utilized to incorporate contextual
information into feature descriptors, enhancing their robustness [13, 27, 28].
Motivated by these advancements, we develop a Transformer-based booster
designed to boost the robustness and accuracy of descriptors by integrating
both the visual and geometric information from all keypoints into individual
descriptors. A noteworthy aspect of our approach is the use of a wave-based
position encoder, which offers enhanced encoding capabilities. Leveraging the
global receptive field in Transformer, the boosted descriptors encapsulate
global contextual information, thereby becoming more robust and
discriminative.

Figure 1:  The visualization of detected keypoints and the matches using our
RADA on the Megadepth [9] validation set. (a) Keypoints detection from the
source domain (daytime). (b) Keypoints detection from the target domain
(nighttime). (c) The matching results of image pairs composed of different
source and target domain images. The green lines show correct correspondences.

In summary, the main contributions of this paper are as follows:

  * -

We introduce domain adaptation supervision to align high-level feature
distributions across different domains and achieve invariant representations
using transfer learning concepts.

  * -

We propose a Transformer-based booster to enhance descriptor robustness by
integrating the global descriptors and positional information through the
concepts of amplitude and phase of waves.

  * -

We develop a hierarchical feature aggregation network incorporating these two
key components with carefully designed loss functions to achieve Robust and
Accurate feature Learning with Domain Adaptation (RADA).

##  2 Related Works

###  2.1 Local Feature Learning

Descriptors were predominantly hand-crafted initially, with SIFT and ORB [10]
being the most representative examples. The advent of deep learning has
significantly advanced the performance of learned local descriptors. Most
patch-based learning descriptors [22] adopt the network architecture
introduced in L2-Net [21], each with different loss functions. However, these
methods primarily focus on descriptor extraction, with their receptive fields
limited to the image patch. Other descriptors [34], following the ’detect-
then-describe’ paradigm (score map-based methods), estimate score maps and
descriptor maps, where the score map indicates keypoint probability.

ALIKE [34] introduces a differentiable keypoint detection module based on the
score map, enabling backpropagation of gradients and subpixel-level keypoint
production. Additionally, joint detection and description methods have shown
increasing performance. SuperPoint [5] proposes a network trained on
homography image pairs. R2D2 [16] computes the repeatability and reliability
maps for keypoint detection. DISK [24] employs reinforcement learning to train
both score and descriptor map. D2-Net [6] detects keypoints using channel and
spatial maxima on low-resolution feature maps. ASLFeat [14] uses multi-level
features to detect keypoints and models the local shape with deformable
convolutions, enhancing localization accuracy and descriptors.

Despite numerous related works focusing on learning local features, there is
limited attention on enhancing descriptor robustness and accuracy through
domain adaptation and the implementation techniques of domain supervision.

###  2.2 Domain Adaptation

Domain adaptation aims to bridge the gap between features learned from
networks across source and target data. This concept has been extensively
developed through deep learning, enhancing adaptation performance in computer
vision and multimedia applications [25, 29]. Several methods [26] employ the
Maximum Mean Discrepancy (MMD) [15] loss, which computes the norm of the
difference between two domain means, to minimize the difference of their
feature distributions. Other approaches [25] motivate from adversarial
learning, with the gradient reversal algorithm (ReverseGrad) from DANN [7]
being particularly relevant. This algorithm treats domain invariance as a
binary classification problem, directly maximizing the loss of the domain
classifier by reversing its gradients to learn a representation that is
simultaneously discriminative of source labels while indistinguishable between
domains.

In local feature learning tasks, DomainFeat [30] is the first to introduce
domain adaptation to the field, developing image-level domain invariance
supervision by fusing domain-invariant representations. Motivated by these
advancements, we implement our domain adaptation supervision, which
organically combines the MMD loss and gradient reversal algorithm to learn
accurate and robust features across different domains.

###  2.3 Feature Context Awareness

The distribution of feature locations and descriptors within an entire image
forms a global context, and incorporating this into descriptors has become a
rising trend in local feature matching methods [19]. ContextDesc [13] encodes
both geometric and visual contexts by using keypoint locations, raw local
features, and high-level regional features as inputs. MTLDesc [27] introduces
an adaptive global context enhancement module and multiple local context
enhancement modules to inject non-local contextual information, thereby
efficiently learning high-level dependencies between distant features.
FeatureBooster [28] only takes the descriptors and geometric information, such
as 2D locations, as inputs and uses a lightweight Transformer to aggregate
them and produce new descriptors.

However, most of these works use MLPs as position encoders, which have
relatively weak encoding abilities. Inspired by Wave-MLP [20, 12], we
recognize that phase information is equally important as amplitude information
in vectors. We encode the descriptor as amplitude information and the position
as phase information, then fuse the visual and geometric information using the
Euler formula to generate position-aware descriptors in the Transformer-based
booster.

##  3 Methodology

Figure 2:  Network architecture. Our RADA consists of three components:
keypoint detection and descriptor extraction backbone to learn accurate
descriptors from hierarchical features, domain adaptation supervision to
achieve domain-invariant representations, and a Transformer-based booster to
enhance the robustness of descriptors. The cross-domain training image pairs
with ground truth are produced from Megadepth [9].

###  3.1 Backbone

The procedure of our network to learn global domain-invariant features
includes feature encoding, feature aggregation, and local feature extraction.
Starting from an input image $I\in\mathbb{R}^{H\times W\times 3}$, the process
proceeds to generate a score map $S\in\mathbb{R}^{H\times W}$ and a descriptor
map $D\in\mathbb{R}^{H\times W\times dim}$, and finally identifies sub-pixel
keypoints $\\{p=[u,v]^{T}\\}$ along with their corresponding descriptors
$\\{d\in\mathbb{R}^{dim}\\}$.

Feature Encoding: The feature encoder transforms the input image $I$ into
multi-scale features $F_{1},F_{2},F_{3},F_{4}$ using four encoding blocks with
channel ranges of 32, 64, 128, and 128. The first block consists of two-layer
3×3 convolutions with ReLU activation that extract low-level image features
$F_{1}$. The last three blocks contain a max-pooling layer and a 3×3 basic
ResNet block. The downsample rate of max-pooling in block 2 is 0.5, and 0.25
in blocks 3 and 4.

Feature aggregation: It combines multi-scale features
$F_{1},F_{2},F_{3},F_{4}$ to enhance both localization and representation
capabilities. We use a 1×1 convolution and a bilinear upsampling layer for the
hierarchical features to align their dimensions and resolutions. By
concatenating these aligned features, we obtain the aggregated feature $F$ for
the next keypoint detection and descriptor extraction.

Feature extraction: This component first produces an ${H\times
W\times(dim+1)}$ feature map, which consists of two main components. The first
dim channels are L2 normalized to form the descriptor map $D$ and the last
channel is normalized using Sigmoid activation to create the score map $S$. A
Differentiable Keypoint Detection (DKD) module [34] is applied to detect
trainable, differentiable keypoints. It first uses non-maximum suppression
(NMS) on the score map $S$ to identify local maxima, and then refines the
positions of these pixel-level keypoints with softargmax on the local patches,
extracting differentiable subpixel keypoints. Based on the accurate keypoints,
the network then samples their corresponding descriptors from the dense
descriptor map $D$. Finally, the keypoints and descriptors are further
enhanced through the Transformer-based booster described in section 3.3,
resulting in accurate local features.

###  3.2 Domain Adaptation Supervision

For the source domain image $I_{S}$ and the transformed target domain image
$I_{T}$, the inconsistency between their feature domains can disrupt the
processing of keypoint detection and descriptor extraction.

As shown in Fig. 3(a), to align the distribution between the high-level
feature maps of $I_{S}$ and $I_{T}$ , we first apply the Maximum Mean
Discrepancy (MMD) metric [15] with a linear kernel function. This distance is
computed with respect to a particular representation, $\phi(\cdot)$, operating
on source domain high-level features, $x_{s}\in X_{S}$, and target domain
high-level features, $x_{t}\in X_{T}$. The empirical approximation of this
distance is computed as follows:

$\mathcal{L}_{\mathrm{MMD}}(X_{S},X_{T})=\left\|\frac{1}{|X_{S}|}\sum_{x_{s}\in
X_{S}}\phi(x_{s})-\frac{1}{|X_{T}|}\sum_{x_{t}\in X_{T}}\phi(x_{t})\right\|.$
(1)

Next, we apply a gradient reversal layer [7] to implement domain adversarial
learning, making the adversarial discriminator indistinguishable between
domains. This is followed by fully connected layers with channels to (dim,
512, 128), culminating in a binary classifier. The domain classifier
distinguishes the global domain-invariant features $(X_{S},X_{T})$ with the
cross-entropy loss:

$\mathcal{L}_{\mathrm{adv}}(X_{S},X_{T})=\frac{1}{N}\sum_{i=1}^{N}\left(-l_{i}\log(s_{i})-(1-l_{i})\log(1-s_{i})\right),$
(2)

where $l_{i}$ denotes the domain category label of the image: $l_{i}=1$ for
the target image $I_{T}$, and $l_{i}=0$ for the source image $I_{S}$. $s_{i}$
represents the domain prediction score of the target image $I_{T}$.

During forward propagation, the features of the gradient reversal layer remain
unchanged, whereas during back propagation, the gradient returned by the
gradient reversal layer maximizes the loss function, updating the parameters
in the direction of the negative gradient. It minimizes the distribution
distance and maximizes the domain classifier loss, resulting in domain-
invariant features. This process weakens the discrimination between the target
and source images for the discriminator network, mapping the feature domains
of the two images as closely as possible, thereby improving network
performance. Finally, we define the loss function of the domain adaptation
supervision as:

$\mathcal{L}_{\mathrm{da}}=\mathcal{L}_{\mathrm{adv}}+\lambda\mathcal{L}_{\mathrm{MMD}}.$
(3)

(a) Domain Adaptation Supervision

(b) Wave Position Encoder

Figure 3:  Clear illustration of the two modules. (a) Domain adaptation is
achieved by adding a domain classifier connected to the feature map via a
gradient reversal layer. (b) Wave-PE fuses the amplitude $A$ estimated with
the descriptor $d$ and the phase $\theta$ estimated with the position $p$ to
generate position encoding.

###  3.3 Transformer-based Booster

Given $N$ detected keypoints in the image $I$, their positions
$p\in\mathbb{R}^{N\times 3}$ and descriptors $d\in\mathbb{R}^{N\times C}$ are
dynamically fused using a Wave Position Encoder (Wave-PE) to enhance
descriptor robustness. The global context, encompassing descriptors of other
features and the spatial layout of all features, is then integrated using a
Transformer. This incorporation of global information significantly enhances
the robustness of individual descriptors, resulting in improved performance
under challenging domain variations.

####  3.3.1 Wave Position Encoder

strengthens descriptors with more information by fusing position and
descriptor information through amplitude and phase relationships. This method
offers a stronger encoding capacity compared to traditional MLP-based position
encoders.

In Wave-PE, position encoding is represented as a wave $\tilde{w}$ with both
amplitude $A$ and phase $\theta$ information. The Euler formula is employed to
decompose waves into real and imaginary parts, allowing efficient processing:

$w_{j}=A_{j}\odot e^{i\theta_{j}}=A_{j}\odot\cos\theta_{j}+i\cdot
A_{j}\odot\sin\theta_{j},\quad j=1,2,\ldots,n.$ (4)

As shown in Fig. 3(b), the amplitude and phase are estimated by two learnable
networks via descriptors and positions, respectively. A learnable network then
fuses the real and imaginary parts into position encoding:

$\displaystyle
A_{j}=\mathrm{MLP}_{A}(d_{j}),\quad\theta_{j}=\mathrm{MLP}_{\theta}(p_{j}),$
(5) $\displaystyle
d^{\mathrm{PE}}_{j}=d_{j}+\mathrm{MLP}_{F}\left([A_{j}\odot\cos\theta_{j},A_{j}\odot\sin\theta_{j}]\right),$

where $[\cdot,\cdot]$ denotes concatenation. For simplicity, a two-layer MLP
is used for the networks in eq. 5.

####  3.3.2 Attention-Free Transformer

(AFT) offers lower time and space complexity compared to the traditional
multi-head attention (MHA) mechanism used in vanilla Transformers. Inspired by
FeatureBooster [28], we use AFT-attention [32] to aggregate all local feature
information, forming a global context that enhances descriptor robustness. AFT
rearranges the computation order of $Q$, $K$, $V$, and then multiplies $K$ and
$V$ element-wise instead of using matrix multiplication. The Attention-Free
Transformer for keypoint $i$ is formulated as:

$\displaystyle f_{i}(X)$
$\displaystyle=\sigma(Q_{i})\odot\frac{\sum_{j=1}^{N}\exp(K_{j})\odot
V_{j}}{\sum_{j=1}^{N}\exp(K_{j})}$ (6)
$\displaystyle=\sigma(Q_{i})\odot\sum_{j=1}^{N}(\mathrm{softmax}(K)\odot
V)_{j},$

where $\sigma(\cdot)$ is a Sigmoid function; $Q_{i}$ represents $i$-th row of
$Q$, and $K_{j}$, $V_{j}$ represent the $j$-th rows of $K$, $V$. AFT performs
a revised version of the MHA operation where the number of attention heads
equals the model’s feature dimension $D$, and the similarity used in MHA is
replaced by a kernel function $sim(Q,K)=\sigma(Q)\cdot softmax(K)$. In this
way, attention can be computed by element-wise multiplication instead of
matrix multiplication, resulting in a time and space complexity that is linear
with context and feature size. Thus, the distinguishability of local features
can be improved.

Following previous work [16, 28], we treat descriptor matching as a nearest
neighbor retrieval problem and use Average Precision (AP) to train the
descriptors [4]. The attention-boosted local feature descriptors,
$\\{d^{PE}_{1},d^{PE}_{2},\ldots,d^{PE}_{N}\\}\xrightarrow{}d^{Tr}_{j}$, are
optimized to maximize AP for all descriptors $d^{Tr}$, with the training goal
to minimize the following cost function:

$\mathcal{L}_{\mathrm{Tr}}=1-\frac{1}{N}\sum_{i=1}^{N}\mathrm{AP}(d^{\mathrm{Tr}}_{j}).$
(7)

###  3.4 The Loss Functions

In addition to the domain adaptation loss and Transformer-based boosting loss
described above, we also define the detector loss, descriptor loss and
coupling loss for the entire supervision training. To supervise the features,
we adopt the reprojection loss and the neural reprojection error (NRE) loss
proposed in [34].

Considering an image pair $(I_{A},I_{B})$, the network extracts score maps
$S_{A}$ and $S_{B}$ from the image pair. The DKD module then detects the
keypoints $P_{A}\in\mathbb{R}^{N_{A}\times 2}$ and
$P_{B}\in\mathbb{R}^{N_{B}\times 2}$, respectively. Corresponding descriptors
for $P_{A}$ and $P_{B}$ are denoted by $D_{A}\in\mathbb{R}^{N_{A}\times dim}$
and $D_{B}\in\mathbb{R}^{N_{B}\times dim}$, respectively. We define the loss
functions as follows:

####  3.4.1 Detector Loss

is defined as the reprojection loss. Since the keypoints extracted are
differentiable, we can directly train their positions using the reprojection
distance. For a keypoint $p_{A}$ in image $I_{A}$, we warp it to image $I_{B}$
using 3D perspective projection:

$p_{AB}=\mathrm{\mathbf{wrap}}_{AB}(p_{A})=\pi(d_{A}R_{AB}\pi^{-1}(p_{A})+t_{AB}),$
(8)

where $R_{AB}$ and $t_{AB}$ denote the rotation and translation matrices from
$I_{A}$ to $I_{B}$, respectively. $d_{A}$ represents the depth of $p_{A}$, and
$\pi(p)$ is the process of projecting a 3D point $P=[X,Y,Z]^{T}$ onto the
image plane. In $I_{B}$, we search for the nearest keypoint $p_{B}$ to
$p_{AB}$, ensuring their distance is less than $th_{gt}$ pixels. This keypoint
$p_{B}$ is considered the matching keypoint of $p_{A}$. Similarly, we project
$p_{B}$ back to $I_{A}$ to obtain $p_{BA}$. The reprojection loss of
$(p_{A},p_{B})$ is:

$\mathcal{L}_{\mathrm{det}}(p_{A},p_{B})=\frac{1}{2}\left(\|p_{A}-p_{BA}\|+\|p_{B}-p_{AB}\|\right).$
(9)

The overall reprojection loss $\mathcal{L}_{det}$ is then calculated as the
average reprojection loss of all matching keypoints in both images.

####  3.4.2 Descriptor Loss

abandons the classical triplet loss and adopts the NRE loss [8] to optimize
the dense descriptor map. It minimizes the cross-entropy difference between
the dense reprojection probability map and the dense matching probability map,
thereby providing a comprehensive constraint for the dense descriptor map and
a stable training process.

For a keypoint $p_{A}$ in image $I_{A}$, and its reprojected keypoint $p_{AB}$
in image $I_{B}$, the reprojection probability map is defined by $p_{AB}$
using the bilinear interpolation as in [34]: $q_{r}\left(p_{A}|p_{AB}\right)$.
For the descriptor $d_{p_{A}}\in\mathbb{R}^{dim}$ of keypoint $p_{A}$ and the
dense descriptor map $D_{B}\in\mathbb{R}^{(H\times W)\times\text{dim}}$, their
matching probability map is given as the softmax normalization:

$q_{m}(p_{B}\mid d_{p_{A}},D_{B}):=\mathrm{softmax}\left(\frac{D_{B}\cdot
d_{p_{A}}-1}{t_{des}}\right),$ (10)

where $t_{des}$ controls the sharpness of matching probability map. The NRE
loss [8] minimizes the difference between the reprojection probability map and
the matching probability map using cross-entropy (CE):

$\displaystyle NRE(p_{A},I_{B})$
$\displaystyle:=\mathrm{CE}\left(q_{r}(p_{B}\mid p_{AB})\parallel
q_{m}(p_{B}\mid d_{p_{A}},D_{B})\right)$
$\displaystyle=-\ln\left(q_{m}(p_{AB}\mid d_{p_{A}},D_{B})\right).$ (11)

Thus, we define the descriptor loss in a symmetric way as:

$\mathcal{L}_{\mathrm{des}}=\frac{1}{N_{A}+N_{B}}\left(\sum_{p_{A}\in
I_{A}}\mathrm{NRE}(p_{A},I_{B})+\sum_{p_{B}\in
I_{B}}\mathrm{NRE}(p_{B},I_{A})\right),$ (12)

where $N_{A}$ and $N_{B}$ are the number of keypoints in image $I_{A}$ and
$I_{B}$, respectively.

####  3.4.3 Coupling Loss

constraints the keypoints in discriminative areas to improve their reliability
with the assistance of descriptors. Inspired by methods following the ’detect-
and-describe’ paradigm like D2-Net [6], we define it as:

$\mathcal{L}_{\mathrm{cp}}^{A}=\frac{1}{N_{A}}\sum_{\begin{subarray}{c}p_{A}\in
I_{A}\\\ p_{AB}\in
I_{B}\end{subarray}}\frac{s_{p_{A}}s_{p_{AB}}}{\sum_{\begin{subarray}{c}p^{\prime}_{A}\in
I_{A}\\\ p^{\prime}_{AB}\in
I_{B}\end{subarray}}s_{p^{\prime}_{A}}s_{p^{\prime}_{AB}}}\mathcal{M}(p_{AB},d_{p_{A}},D_{B}),$
(13)

where $N_{A}$ is the number of keypoints in the image $I_{A}$. For each
keypoint $p_{A}$ in image $I_{A}$, $p_{AB}$ is its corresponding projection
keypoint in image $I_{B}$. $s_{p}$ denotes the score value of keypoint $p$ in
its corresponding image. Specially, we define the ranking loss
$\mathcal{M}(\hskip 0.4pt)$ for representation learning as:

$\mathcal{M}(p_{AB},d_{p_{A}},D_{B})=1-\mathcal{S}\left(\exp\left(\frac{D_{B}\cdot
d_{p_{A}}-1}{t_{des}}\right),p_{AB}\right),$ (14)

where $\mathcal{S}(\cdot,\cdot)$ is the bilinear sampling at position
$\in\mathbb{R}^{2}$ and probability map $\in\mathbb{R}^{H\times W}$.
Similarly, the overall coupling loss is given symmetrically as:

$\mathcal{L}_{\mathrm{cp}}=\frac{1}{2}\left(\mathcal{L}_{\mathrm{cp}}^{A}+\mathcal{L}_{\mathrm{cp}}^{B}\right).$
(15)

####  3.4.4 Total Loss

combines these loss functions, each with its own coefficient:

$\mathcal{L}_{\mathrm{total}}=\omega_{\mathrm{da}}\mathcal{L}_{\mathrm{da}}+\omega_{\mathrm{Tr}}\mathcal{L}_{\mathrm{Tr}}+\omega_{\mathrm{det}}\mathcal{L}_{\mathrm{det}}+\omega_{\mathrm{des}}\mathcal{L}_{\mathrm{des}}+\omega_{\mathrm{cp}}\mathcal{L}_{\mathrm{cp}}.$
(16)

##  4 Experiments

###  4.1 Implementation Details

####  4.1.1 Training Data:

We trained our RADA on MegaDepth [9] and adopted the training scenes used in
DISK [24]. We followed the cross-domain data preparation strategy proposed in
[30], using HiDT [2] to translate the source domain images $I_{S}$ to dusk and
evening image domains $I_{T}$. Since the target domain images also come from
MegaDepth, they correspond to images with ground truth correspondence. We then
computed the overlap score between two images following D2-Net [6] and sampled
100 training pairs with an overlap score in the range of $[0.3,1]$ for each
scene at every epoch.

####  4.1.2 Training Details:

The radius of the differential keypoints detection part is set to 2 pixels,
and keypoints with reprojection distances of less than 5 pixels are considered
ground truth keypoint pairs during training. The coefficients of the overall
loss eq. 16, $(\omega_{da},\omega_{Tr},\omega_{det},\omega_{des},\omega_{cp})$
are optimized by Optuna [1] as $(2,1,1,5,1)$, and $\lambda$ in eq. 3 is set as
0.01. The images were cropped and resized to $480\times 480$ during training.
The network was trained using the ADAM optimizer, with the learning rate
starting at zero and warming up to $3e^{-3}$ over $500$ steps before remaining
constant. We set the batch size to 2 and accumulated gradients over 16
batches. Under these settings, the proposed model converges on a single NVIDIA
RTX 4090 in about 30 hours.

Figure 4:  Visualization of matches on Aachen Day-Night [33]. The inliers are
plotted in green if they are correct (0 to 5 pixels in reprojection error), in
red if they are incorrect (above 5 pixels), and in blue if ground truth depth
is not available.

###  4.2 Image Matching

####  4.2.1 Experiment setup:

HPatches [3] is a common evaluation dataset for image matching, including 116
scenes and 580 image pairs, which exhibit significant changes in illumination
or viewpoints. This dataset provides ground truth homography between each
image pair for evaluation. Following the protocol of D2-Net, we exclude 8
high-resolution sequences from the 116 available sequences and use 108
sequence scenes with viewpoint or illumination changes for a fair comparison.
We record the mean matching accuracy (MMA), the average percentage of correct
matches to all estimated putative matches, under thresholds varying from one
to ten pixels, along with the numbers of features and matches. Specially, We
evaluate the MMA@3 without loss of generality. We also record GFLOPs (Giga
FLoating-point OPerations) and the interface FPS (Frames Per Second), which
represent the computing complexity and response speed. Like D2-Net, we use
mutual nearest neighbor search as the matching method.

Table 1: GFLOPs and FPS on 640 × 480 images, illumination and overall MMA within one to three pixels thresholds of different methods on HPatches [3]. The top three results are marked with red, green, blue. * represents similar results. Models | GFLOPs | FPS | MMA_i@1 | MMA_i@2 | MMA_i@3 | MMA@1 | MMA@2 | MMA@3
---|---|---|---|---|---|---|---|---
D2-Net [6] | 889.40 | 6.60 | 13.76% | 29.01% | 43.22% | 9.78% | 23.52% | 37.29%
SuperPoint [5] | 26.11 | 45.87 | 43.09% | 59.39% | 69.25% | 34.27% | 54.94% | 66.37%
R2D2 [16] | 464.55 | 8.70 | 37.60% | 66.08% | 80.78% | 33.31% | 62.17% | 75.77%
ASLFeat [14] | 44.24 | 8.96 | 46.12% | 64.46% | 75.45% | 39.16% | 61.07% | 72.44%
DISK [24] | 98.97 | 15.73 | 51.98% | 77.11% | 82.61% | 43.71% | 66.98% | 77.59%
ALIKE [34] | 67.63* | 29.15 | 51.24% | 69.51% | 77.40% | 44.97% | 66.21% | 74.51%
RADA | 67.74* | 24.56 | 52.13% | 70.83% | 78.65% | 45.03% | 67.17% | 76.24%

####  4.2.2 Results:

Table 1 shows the results on HPatches under illumination and overall change.
Our method generally outperforms other proposed methods. Although R2D2 [16]
and DISK [24] sometimes perform better on different evaluation metrics, they
sacrifice either the computing complexity or the response speed. Our RADA
balances accracy, computing complexity, and response speed organically,
resulting in better overall performance.

Table 2: Ablation studies of network architecture and losses. DET, DES, and CP
denote detector loss, descriptor loss and coupling loss, respectively.

Methods | MMA@3
---|---
Backbone | 70.48%
\+ Domain Adaptation Supervision | 74.72%
\+ Transformer-based Booster | 76.24%

DET | DES | CP | MMA@3
---|---|---|---
| ✓ | ✓ | 71.86%
✓ | ✓ |  | 68.92%
✓ | ✓ | ✓ | 76.24%

####  4.2.3 Ablation:

We conduct ablation studies on HPatches to explore the performance of our two
special components and architecture loss functions. We evaluate the MMA@3
without loss of generality as shown in Table 2. The domain adaptation
supervision ensures the network learns domain-invariant features, and the
Transformer-based booster enhances the accuracy and robustness of features,
consistent with the ablation study outcomes. The detector loss directly
optimizes keypoints positions to produce accurate keypoints, while the
coupling loss ensures that the keypoints are located in areas where the
descriptors are discriminative. More reliable keypoints produce fewer false
matches, resulting in a higher matching accuracy.

###  4.3 Visual Localization

####  4.3.1 Experiment setup:

We evaluate our method in visual localization, a more complete pipeline in
computer vision. We select an outdoor dataset with severe illumination
changes, Aachen Day-Night dataset v1.1[33], which contains 6697 day-time
database images and 1015 query images (824 for the day and 191 for the night).
We use the hierarchical localization toolbox (HLoc)[18] by replacing the
feature extraction module with different feature detectors and descriptors. We
use the evaluation protocol on the Long-Term Visual Localization Benchmark[23]
and report the percentage of correct localized query images under given error
thresholds. Note that all other methods use mutual nearest neighbor search for
matching. We utilize ratio test or distance test for mutual nearest neighbor
matching, whose thresholds are same as setting in [28].

Table 3: Visual localization results in Aachen Day-Neight [33]. The positional
and angular performances are present (the larger the better). The top two
results are marked with red, green.

Method | Day | Night
---|---|---
(0.25m,2∘) / (0.50m,5∘) / (5.0m,10∘)
SIFT[10] | 87.1 / 93.8 / 98.1 | 50.8 / 70.2 / 81.2
SuperPoint[5] | 87.9 / 94.3/ 98.2 | 67.0 / 84.8 / 95.8
SOSNet[22] |  88.7 / 94.7 / 98.7 | 58.1 / 78.5 / 92.7
R2D2[16] | 86.8 / 94.1 / 98.4 | 57.1 / 76.4 / 88.5
DISK[24] |  86.9 /93.3 / 99.2 |  69.4 /86.8 / 96.7
ALIKE[34] | 87.3 / 93.2 / 98.7 |  67.5 / 85.3 / 97.9
RADA |  87.9 / 94.4 / 99.0 |  72.9 / 86.8 / 98.6

####  4.3.2 Results:

The results are shown in Table 3. This demonstrates that our descriptor based
on domain adaptation has significantly improved the performance of the
descriptor and is efficiently combined with keypoints detection under the day-
night domain changes. As shown in Fig. 4, our method can detect more accurate
keypoints and generate effective matches under the same conditions, and the
descriptors of keypoint locations have higher discriminability. Thus, learning
descriptors with domain adaptation can effectively increase the accuracy of
challenging visual localization tasks.

##  5 Conclusion

In this study, we develop RADA, a multi-level feature aggregation network
incorporating domain adaptation supervision and Transformer-based booster, to
learn robust and accurate features under extreme domain shifts. The
hierarchical network efficiently captures diverse features to achieve accurate
keypoint detection and descriptor extraction. By aligning the high-level
feature distributions across different domains, the domain adaptation
supervision ensures the learning of invariant domain representations. The
Transformer-based booster integrating visual and geometric information is
proposed to enhance the robustness of descriptors. To ensure the accuracy and
robustness of features, we propose three additional losses to apply targeted
supervision on keypoint detection, descriptor extraction, and their coupled
processing. Comprehensive experiments conducted in image matching and visual
localization tasks validate the superiority of our method in these
applications.

## References

  * [1] Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 2623–2631 (2019)
  * [2] Anokhin, I., Solovev, P., Korzhenkov, D., Kharlamov, A., Khakhulin, T., Silvestrov, A., Nikolenko, S., Lempitsky, V., Sterkin, G.: High-resolution daytime translation without domain labels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7488–7497 (2020)
  * [3] Balntas, V., Lenc, K., Vedaldi, A., Mikolajczyk, K.: Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5173–5182 (2017)
  * [4] Boyd, K., Eng, K.H., Page, C.D.: Area under the precision-recall curve: point estimates and confidence intervals. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13. pp. 451–466. Springer (2013)
  * [5] DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 224–236 (2018)
  * [6] Dusmanu, M., Rocco, I., Pajdla, T., Pollefeys, M., Sivic, J., Torii, A., Sattler, T.: D2-net: A trainable cnn for joint detection and description of local features. arXiv preprint arXiv:1905.03561 (2019)
  * [7] Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. In: International conference on machine learning. pp. 1180–1189. PMLR (2015)
  * [8] Germain, H., Lepetit, V., Bourmaud, G.: Neural reprojection error: Merging feature learning and camera pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 414–423 (2021)
  * [9] Li, Z., Snavely, N.: Megadepth: Learning single-view depth prediction from internet photos. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2041–2050 (2018)
  * [10] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 91–110 (2004)
  * [11] Lu, X., Du, S.: Raising the ceiling: Conflict-free local feature matching with dynamic view switching. In: Proceedings of the European Conference on Computer Vision (ECCV) (2024)
  * [12] Lu, X., Yan, Y., Kang, B., Du, S.: Paraformer: Parallel attention transformer for efficient feature matching. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1853–1860 (2023)
  * [13] Luo, Z., Shen, T., Zhou, L., Zhang, J., Yao, Y., Li, S., Fang, T., Quan, L.: Contextdesc: Local descriptor augmentation with cross-modality context. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2527–2536 (2019)
  * [14] Luo, Z., Zhou, L., Bai, X., Chen, H., Zhang, J., Yao, Y., Li, S., Fang, T., Quan, L.: Aslfeat: Learning local features of accurate shape and localization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6589–6598 (2020)
  * [15] Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence, N.: Covariate shift and local learning by distribution matching. Dataset Shift in Machine Learning pp. 131–160 (2008)
  * [16] Revaud, J., De Souza, C., Humenberger, M., Weinzaepfel, P.: R2d2: Reliable and repeatable detector and descriptor. Advances in neural information processing systems 32 (2019)
  * [17] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: 2011 International conference on computer vision. pp. 2564–2571. Ieee (2011)
  * [18] Sarlin, P.E., Cadena, C., Siegwart, R., Dymczyk, M.: From coarse to fine: Robust hierarchical localization at large scale. In: CVPR (2019)
  * [19] Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4938–4947 (2020)
  * [20] Tang, Y., Han, K., Guo, J., Xu, C., Li, Y., Xu, C., Wang, Y.: An image patch is a wave: Phase-aware vision mlp. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10935–10944 (2022)
  * [21] Tian, Y., Fan, B., Wu, F.: L2-net: Deep learning of discriminative patch descriptor in euclidean space. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [22] Tian, Y., Yu, X., Fan, B., Wu, F., Heijnen, H., Balntas, V.: Sosnet: Second order similarity regularization for local descriptor learning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11016–11025 (2019)
  * [23] Toft, C., Maddern, W., Torii, A., Hammarstrand, L., Stenborg, E., Safari, D., Okutomi, M., Pollefeys, M., Sivic, J., Pajdla, T., et al.: Long-term visual localization revisited. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(4), 2074–2088 (2020)
  * [24] Tyszkiewicz, M., Fua, P., Trulls, E.: Disk: Learning local features with policy gradient. Advances in Neural Information Processing Systems 33, 14254–14265 (2020)
  * [25] Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7167–7176 (2017)
  * [26] Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., Darrell, T.: Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  * [27] Wang, C., Xu, R., Zhang, Y., Xu, S., Meng, W., Fan, B., Zhang, X.: Mtldesc: Looking wider to describe better. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 2388–2396 (2022)
  * [28] Wang, X., Liu, Z., Hu, Y., Xi, W., Yu, W., Zou, D.: Featurebooster: Boosting feature descriptors with a lightweight neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7630–7639 (2023)
  * [29] Xia, H., Zhao, H., Ding, Z.: Adaptive adversarial network for source-free domain adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9010–9019 (2021)
  * [30] Xu, R., Wang, C., Xu, S., Meng, W., Zhang, Y., Fan, B., Zhang, X.: Domainfeat: Learning local features with domain adaptation. IEEE Transactions on Circuits and Systems for Video Technology 34(1), 46–59 (2023)
  * [31] Xu, S., Chen, S., Xu, R., Wang, C., Lu, P., Guo, L.: Local feature matching using deep learning: A survey. Information Fusion 107, 102344 (2024)
  * [32] Zhai, S., Talbott, W., Srivastava, N., Huang, C., Goh, H., Zhang, R., Susskind, J.: An attention free transformer. arXiv preprint arXiv:2105.14103 (2021)
  * [33] Zhang, Z., Sattler, T., Scaramuzza, D.: Reference pose generation for long-term visual localization via learned features and view synthesis. International Journal of Computer Vision 129(4), 821–844 (2021)
  * [34] Zhao, X., Wu, X., Miao, J., Chen, W., Chen, P.C., Li, Z.: Alike: Accurate and lightweight keypoint detection and descriptor extraction. IEEE Transactions on Multimedia 25, 3101–3112 (2022)