# Unveiling the Potential of Deep Learning Models for Solar Flare Prediction
in Near-Limb Regions

Chetraj Pandey, Rafal A. Angryk, Berkay Aydin  Department of Computer Science,
Georgia State University, Atlanta, GA, USA
{cpandey1, rangryk<EMAIL_ADDRESS>

###### Abstract

This study aims to evaluate the performance of deep learning models in
predicting $\geq$M-class solar flares with a prediction window of 24 hours,
using hourly sampled full-disk line-of-sight (LoS) magnetogram images,
particularly focusing on the often overlooked flare events corresponding to
the near-limb regions (beyond $\pm$70∘ of the solar disk). We trained three
well-known deep learning architectures–AlexNet, VGG16, and ResNet34 using
transfer learning and compared and evaluated the overall performance of our
models using true skill statistics (TSS) and Heidke skill score (HSS) and
computed recall scores to understand the prediction sensitivity in central and
near-limb regions for both X- and M-class flares. The following points
summarize the key findings of our study: (1) The highest overall performance
was observed with the AlexNet-based model, which achieved an average
TSS$\sim$0.53 and HSS$\sim$0.37; (2) Further, a spatial analysis of recall
scores disclosed that for the near-limb events, the VGG16- and ResNet34-based
models exhibited superior prediction sensitivity. The best results, however,
were seen with the ResNet34-based model for the near-limb flares, where the
average recall was approximately 0.59 (the recall for X- and M-class was 0.81
and 0.56 respectively) and (3) Our research findings demonstrate that our
models are capable of discerning complex spatial patterns from full-disk
magnetograms and exhibit skill in predicting solar flares, even in the
vicinity of near-limb regions. This ability holds substantial importance for
operational flare forecasting systems.

###### Index Terms:

deep learning, solar flares, near-limb prediction

This is a preprint accepted at the 22nd International Conference on Machine
Learning and Applications (ICMLA), 2023. ©IEEE.

##  I Introduction

Solar flares are temporary occurrences on the Sun, considered to be the
central phenomena in space weather forecasting, manifested as the sudden large
eruption of electromagnetic radiation on the outermost atmosphere of the Sun.
They are observed by the X-ray sensors on Geostationary Operational
Environmental Satellite (GOES) and classified according to their peak X-ray
flux level (measured in watt per square meter) into the following five
categories by the National Oceanic and Atmospheric Administration (NOAA): X
$(\geq 10^{-4}Wm^{-2})$, M $(\geq 10^{-5}$ and $<10^{-4}Wm^{-2})$, C $(\geq
10^{-6}$ and $<10^{-5}Wm^{-2})$, B $(\geq 10^{-7}$ and $<10^{-6}Wm^{-2})$, and
A $(\geq 10^{-8}$ and $<10^{-7}Wm^{-2})$ [1]; areas on the Sun with peak X-ray
flux less than $<10^{-8}Wm^{-2}$ (less than A-class flares) are regarded as
flare-quiet (FQ) regions. Large flares (M- and X-class) are rare events and
significantly more powerful than other flare classes, with the capability to
disrupt several infrastructures in space (e.g., satellite operations) and on
Earth (e.g., the electricity power grid and avionics). Hence, a precise and
reliable system for solar flare prediction is essential.

Active regions (ARs) on the Sun are areas with localized magnetic
disturbances, visually indicated by scattered flags in the full-disk
magnetogram image shown in Fig. 1. In most operational flare forecasting
systems, these ARs are used as regions of interest because they are considered
to be the main initiators of space weather events. In AR-based flare
prediction, the underlying models issue predictions for each AR individually.
To issue a full-disk forecast with an AR-based model, the output flare
probabilities for each active region are usually aggregated using a heuristic
function as mentioned in [2]. This heuristic function assumes conditional
independence among ARs and equal contributions from all ARs to the aggregated
full-disk forecast. However, this uniform weighting scheme may not accurately
represent the actual impact of each AR on the probability of full-disk flare
prediction [3].

Figure 1: An annotated full-disk line-of-sight magnetogram observed on
2013-01-09 at 00:00:00 UTC as an example, showing the approximate central
location (within $\pm$70∘) and near-limb (beyond $\pm$70∘ to $\pm$90∘) region
with all the NOAA active regions (ARs) present at the noted timestamp. ARs in
central and near-limb regions are indicated by blue and red flags
respectively. Note that the directions East (E) and West (W) are reversed in
solar coordinates.

Furthermore, the magnetic field measurements, which are the dominant feature
employed by the AR-based forecasting techniques, are susceptible to severe
projection effects [4] as ARs get closer to limbs (to such an extent that
magnetic field readings become distorted beyond $\pm$60∘ of the solar-disk
[5]). Therefore, the aggregated full-disk flare probability is in fact,
limited to ARs in central locations. This further underscores the inherent
challenges in issuing a full-disk flare forecast using an AR-based model. As
studies in AR-based model for flare prediction include ARs located within
$\pm$30∘ [6] to $\pm$70∘ [7], in the context of this paper, this upper limit
($\pm$70∘) is used as a boundary for central location (within $\pm$70∘) and
near-limb regions (beyond $\pm$70∘) as shown in Fig. 1. Full-disk models, on
the other hand, typically use compressed images derived from the original
full-depth magnetogram rasters representing the entire solar disk. While
projection effects persist in these magnetograms, which contain actual
magnetic field readings, the compressed images retain only the shape-based
spatial patterns of AR such as size, directionality, sunspot borders, and
inversion lines in grayscale. Thus, by incorporating the entire full-disk
magnetogram, this approach enables the prediction of solar flares in the Sun’s
near-limb areas as well, which are often overlooked by AR-based methods.

To the best of our knowledge, flare prediction employs four major approaches:
(i) empirical human prediction (e.g., [8]), (ii) statistical prediction (e.g.,
[9]), (iii) physics-based numerical simulations (e.g., [10]), and (iv) machine
learning and deep learning approaches (e.g., [11, 7, 12, 13, 14, 15]). The use
of machine learning in extracting forecast patterns from the Sun has been an
active area of research since the early 1990s [16]. Since then, there has been
a significant advancement in machine learning and deep learning techniques,
leading to a surge of interest in applying these methods to build more precise
flare forecasting models as they can automatically extract relevant spatial
features discerning the intricate structures associated with ARs prone to
eventual solar flares.

In [6], a convolutional neural network (CNN) model was trained using solar
Active Region (AR) patches extracted from line-of-sight (LoS) magnetograms
within $\pm$30∘ of the central meridian to predict $\geq$C-, $\geq$M-, and
$\geq$X-class flares. Similarly, [11] developed a CNN-based model that issued
binary class predictions for $\geq$C- and $\geq$M-class flares within 24 hours
using Space-Weather Helioseismic and Magnetic Imager Active Region Patches
(SHARP) data [17]. The SHARP data was extracted from solar magnetograms using
AR patches located within $\pm 45^{\circ}$ of the central meridian. Notably,
both models [6, 11] had limited operational capability, as they were
restricted to small portions of the observable disk in central locations ($\pm
30^{\circ}$ and $\pm 45^{\circ}$). More recently, we presented full-disk
models trained with limited data in [18, 19]. However, these were preliminary
studies that did not provide insights into the model’s capability for near-
limb events.

Moreover, our prior work on explainable full-disk deep learning models [20,
13] shows that the features learned by these models are linked to relevant ARs
on full-disk magnetograms which underscore their possible implications as a
complementary approach. In this study, we present a more comprehensive view of
deep learning-based full-disk models for predicting $\geq$M-class solar flares
in binary mode. We evaluate and compare the performance of three widely used
pre-trained CNNs – AlexNet [21], VGG16 [22], and ResNet34 [23] – by utilizing
hourly sampled instances of full-disk LoS magnetogram images covering solar
cycle 24. The focus of this work is to study whether our models can be relied
upon for critical applications, particularly for near-limb forecasting with
quantitative spatial analytics. The novel contributions of this paper are as
follows: (i) We show an improved overall performance of a full-disk solar
flare prediction model by building and comparing three CNN architectures on
full-disk magnetogram images, (ii) We provide an extended spatial analysis of
the predictive capability of full-disk models on near-limb and central
locations, and (iii) We provide results that underscore the role of full-disk
models in the prediction of solar flares in near-limb regions of the Sun.

The remainder of the paper is structured as follows. In Sec. II, we detail the
process of data preparation with consequent data distribution. In Sec. III, we
describe all three model architectures explored in this work. Sec. IV presents
the experimental design and model evaluation with skill scores and provides
models’ prediction sensitivity in central and near-limb regions. Finally, in
Sec. V, we summarize our findings and suggest avenues for future research.

##  II Data

Figure 2: An illustration of data labeling process for hourly observations of
full-disk LoS magnetogram images with a prediction window of 24 hours. Here,
‘FL’ and ‘NF’ indicate ‘Flare’ and ‘No Flare’ classes. The gray-filled circles
indicate hourly spaced timestamps for magnetogram instances. Figure 3: Data
distribution of four tri-monthly partitions for predicting $\geq$M1.0-class
flares. Note that the length of the bars are in logarithmic scale.

We use compressed images of full-disk LoS solar magnetograms obtained from the
Helioseismic and Magnetic Imager (HMI) [24] instrument onboard Solar Dynamics
Observatory (SDO) [25], publicly available from Helioviewer111Helioviewer :
https://api.helioviewer.org/. We collected a total of 63,649 magnetogram
images by sampling them every hour of the day, starting at 00:00 and ending at
23:00, from December 2010 to December 2018. These images are resized to
512$\times$512 for computational efficiency and labeled using a 24-hour
prediction window based on the maximum peak X-ray flux (converted to NOAA
flare classes), as illustrated in Fig. 2. To elaborate, if the maximum X-ray
intensity of a flare was weaker than M, we labeled the observation as ”No
Flare” (NF: $<$M), and if it was $\geq$M, we labeled it as ”Flare” (FL:
$\geq$M). This resulted in 54,649 instances for the NF class and 9,000
instances (8,120 instances of M-class and 880 instances of X-class flares) for
the FL class. Due to scarce M- and X-class flares, the overall class imbalance
in our dataset is $\sim$1:6 (FL:NF). Finally, we created a non-chronological
split of our data into four temporally non-overlapping tri-monthly partitions
introduced in [18] for our 4-fold cross-validation experiments, where three of
the partitions are used as a training set, and one partition is used as a test
set. The detailed class-wise data distribution is shown in Fig. 3

##  III Models

In this work, we use three CNN architectures: AlexNet [21], VGG16 [22], and
ResNet34 [23]. We used AlexNet [21] due to its inherent architectural
simplicity, consisting of 5 convolutional layers, 3 max pool layers, 1
adaptive average pool layer, and three fully connected layers. Moreover, our
study included VGG16 [22], a relatively more complex model, to evaluate the
hypothesis that an increase in the number of layers might engender enhanced
performance. This model augments the foundational structure of AlexNet by
integrating additional convolutional layers, all employing uniform 3x3
convolutional kernels. The VGG16 architecture consists 13 convolutional
layers, 5 max pool layers, 1 adaptive average pool layer, and 3 fully
connected layers. Lastly, we included ResNet34 [23], a CNN model that extends
the complexity of the VGG16 design by facilitating the training of deeper
networks with fewer parameters. Diverging from the approach of AlexNet and
VGG16, ResNet34 integrates residual connections from each layer into
subsequent connected layers. The architecture of ResNet34 consists of 33
convolutional layers, including a 7x7 kernel for the initial layer and 3x3
kernels for the remaining layers, along with one max pool layer, one adaptive
average pool layer, and one fully connected layer.

The primary reason behind our choice of these three architectures was to
analyze and evaluate the influence of varying architectural designs and
increasing layer depths on performance. Additionally, we factored the
simplicity of the architectures into our selection process, in light of the
relatively modest scale of our dataset for deep learning models. These pre-
trained models require a 3-channel image for input, however, our data
comprises compressed solar magnetogram images, which are grayscale. To
reconcile this, we incorporated an additional convolutional layer at the onset
of the network architecture, which accepts a 1-channel input. This layer
employs a 3$\times$3 kernel with a size-1 stride, padding, and dilation, and
consequently generates a 3-channel feature map. We initialize the added
convolutional layer in all three models using Kaiming initialization [26],
while all other layers are initialized with pre-trained weights.

##  IV Experimental Evaluation

###  IV-A Experimental Settings

We trained our full-disk flare prediction models using Stochastic Gradient
Descent (SGD) as the optimizer and Negative Log-Likelihood (NLL) as the
objective function. We initialized each of the models with their corresponding
pre-trained weights, then further trained it for 50 epochs while employing a
dynamic learning rate scheduling strategy, OneCycleLR with cosine annealing
[27]. The initial learning rate (LR) employed for all three models was $1e-5$,
with a maximum LR set to $1e-5$ for VGG16 and ResNet34, and $1e-4$ for the
AlexNet model. This scheduler adjusts the learning rate in a cyclical pattern,
gradually increasing it to help the model quickly converge and then decreasing
it to fine-tune the performance. The steps per epoch were set to the number of
batches in training data, and the batch size was 64. Using the OneCycleLR
scheduler simplifies and optimizes the models’ learning rate schedule for
hyperparameter tuning.

As mentioned earlier in Sec. II, our dataset has an inherent class imbalance
issue. This imbalance can significantly influence the performance of the
models, potentially leading to less precise and reliable predictions for the
minority class. To address this, we employed data augmentation and adjusted
class weights in the loss function only on the training set. Specifically, we
applied three augmentation techniques: vertical flipping, horizontal flipping,
and rotations between +5∘ and -5∘ to both classes. For each instance in the
minority class (FL), we applied all three augmentations, quadrupling the total
number of instances for the entire FL-class. For each instance in NF-class, we
randomly selected one of the three aforementioned augmentation techniques,
doubling the total instances for this class. The goal of augmenting the NF-
class instances was to ensure that the NF-class, though not uniformly
augmented, retained a diversity in its data akin to the FL-class and expanded
the overall dataset. Post augmentation, we adjusted the class weights to be
inversely proportional to class frequencies, thereby penalizing
misclassifications of the minority class. Finally, we evaluated our models
using a 4-fold cross-validation approach on tri-monthly partitions.

We evaluate the overall performance of our models using two widely-used
forecast skills scores: true skill statistic (TSS, in Eq. 1) and Heidke skill
score (HSS, in Eq. 2), derived from the elements of confusion matrix: True
Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives
(FN). In the context of this paper, the FL class is considered as the positive
outcome, while NF is negative. Lastly, we report the subclass and overall
recall (shown in Eq. 3) for flaring instances (M- and X-class) to assess the
prediction sensitivity of our models in central and near-limb regions.

$TSS=\frac{TP}{TP+FN}-\frac{FP}{FP+TN}$ (1)

$HSS=2\times\frac{TP\times TN-FN\times FP}{((P\times(FN+TN)+(TP+FP)\times
N))},$ (2)

where $N=TN+FP$ and $P=TP+FN$.

$Recall=\frac{TP}{TP+FN}\vspace{2pt}$ (3)

TSS and HSS values range from -1 to 1, where 1 indicates all correct
predictions, -1 represents all incorrect predictions, and 0 represents no
skill. In contrast to TSS, HSS is an imbalance-aware metric, and it is common
practice to use HSS in combination with TSS for the solar flare prediction
models due to the high class-imbalance ratio present in the datasets.

###  IV-B Evaluation

This section presents an analysis of the results, focusing on the performance
comparison of the three models used in this study. The findings reveal that
the AlexNet-based model exhibits better performance in relation to both the
VGG16- and ResNet34-based models, as evidenced by the HSS and TSS scores
provided in Table I. Notably, the AlexNet-based model demonstrates enhanced
robustness, as indicated by the lower standard deviation, and achieves an
approximate 2% improvement (for both TSS and HSS) compared to the VGG16-based
model. Furthermore, when compared to the ResNet34-based model, the AlexNet-
based model showcases a 1% higher skill score (for both TSS and HSS). It is
important to highlight that the skill scores of the VGG16 and ResNet34 models
exhibit greater variability, primarily influenced by the outcomes of Fold-3 in
the 4-fold cross-validation experiment, details presented in Table. II.
Furthermore, our best results exceed those reported in [13, 20] by
approximately 2% in terms of TSS (reported $\sim$0.51), and in [20] by about
2% in terms of HSS ($\sim$0.35), while remaining comparable to the HSS in
[13]. For reproducibility, source code for models along with results is
available in our open-source project repository [28].

TABLE I: The aggregated results in terms of TSS and HSS from test sets of a 4-fold cross-validation experiment on our models. Models |  TSS |  HSS
---|---|---
AlexNet |  0.526$\pm$0.05 |  0.372$\pm$0.05
VGG16 |  0.506$\pm$0.09 |  0.353$\pm$0.09
ResNet34 |  0.513$\pm$0.09 |  0.360$\pm$0.09
TABLE II: Detailed results of 4-fold cross-validation experiments, showing all the four outcomes of confusion matrices (TP, FP, TN, FN) for all three models used in this study. Models | Folds | TP | FP | TN | FN | TSS | HSS
---|---|---|---|---|---|---|---
| Fold-1 | 1,729 | 2,225 | 10,229 | 605 | 0.5621 | 0.4385
AlexNet | Fold-2 | 1,075 | 2,298 | 11,557 | 537 | 0.5010 | 0.3380
| Fold-3 | 1,660 | 3,291 | 11,017 | 704 | 0.4722 | 0.3241
| Fold-4 | 2,209 | 3,549 | 10,483 | 481 | 0.5683 | 0.3890
| Fold-1 | 1,704 | 2,067 | 10,387 | 630 | 0.5641 | 0.4512
VGG16 | Fold-2 | 1,233 | 3,401 | 10,454 | 379 | 0.5194 | 0.2841
| Fold-3 | 1,409 | 3,089 | 11,219 | 955 | 0.3801 | 0.2761
| Fold-4 | 2,125 | 3,236 | 10,796 | 565 | 0.5593 | 0.3992
| Fold-1 | 1,779 | 2,145 | 10,309 | 555 | 0.5900 | 0.4621
ResNet34 | Fold-2 | 1,257 | 3,872 | 9,983 | 355 | 0.5003 | 0.2550
| Fold-3 | 1,328 | 2,299 | 12,009 | 1,036 | 0.4011 | 0.3280
| Fold-4 | 2,160 | 3,382 | 10,650 | 530 | 0.5620 | 0.3933
Figure 4: A heatmap illustrating all three models’ recall performance for $\geq$M-class flares i.e., FL-class. The locations of the flares (with maximum peak x-ray flux, used as labels) are aggregated into 5∘ $\times$ 5∘ spatial bins of latitude and longitude. Note: White cells in the grid represent unavailable instances. TABLE III: A comprehensive results showing correctly (TP) and incorrectly (FN) classified X- and M-class flares, distinguishing between central ($|longitude|$$\leq\pm 70^{\circ}$) and near-limb locations. All counts are aggregated across the folds from the test set of the cross-validation experiment and Recall statistics are computed on the aggregated results. |  | Within $\pm$70∘ | Beyond $\pm$70∘
---|---|---|---
Models | Flare-Class | TP | FN | Recall | TP | FN | Recall
| X-Class | 614 | 54 | 0.92 | 138 | 74 | 0.65
AlexNet | M-Class | 4,645 | 1,185 | 0.80 | 1,276 | 1,014 | 0.55
| Total (X&M) | 5,259 | 1,239 | 0.81 | 1,414 | 1,088 | 0.57
| X-Class | 560 | 108 | 0.84 | 165 | 47 | 0.78
VGG16 | M-Class | 4,473 | 1,357 | 0.77 | 1,273 | 1,017 | 0.55
| Total (X&M) | 5,033 | 1,465 | 0.77 | 1,438 | 1,064 | 0.57
| X-Class | 612 | 56 | 0.92 | 172 | 40 | 0.81
ResNet34 | M-Class | 4,449 | 1,381 | 0.76 | 1,291 | 999 | 0.56
| Total (X&M) | 5,061 | 1,437 | 0.78 | 1,463 | 1,039 | 0.59
|
---|---
(a) | (b)
Figure 5: Individual heatmaps illustrating all three models’ recall
performance for subclasses in FL (a) X-class flares and (b) M-class flares.
The spatially aggregated recall scores in Fig.4 are isolated for two
subclasses. White cells in the grid represent unavailable instances.

In addition, we evaluate the results by examining the correct prediction and
missed flare counts for class-specific flares (X-class and M-class) in central
locations and near-limb locations of the Sun, as presented in Table III. It is
noteworthy that, while the overall performance measured in terms of TSS and
HSS indicates the better performance of the AlexNet-based model over the other
two deeper and more advanced models, VGG16 and ResNet34, the ResNet34-based
model exhibits the best performance on average for near-limb events. The
class-specific analysis for X- and M-class flares reveals that the
ResNet34-based model achieves correct predictions for $\sim$81% of the X-class
flares ($\sim$16% higher than AlexNet) and $\sim$56% of the M-class flares
($\sim$1% higher than AlexNet) in near-limb locations. Despite all models
being fine-tuned with the same dataset and undergoing similar hyperparameter
optimization, their distinctive architectures influenced their ability to
capture specific spatial patterns and features, leading to variations in
overall performance, prediction sensitivity, and recall rates for specific
flare intensities and event locations.

Moreover, we scrutinized the proficiency of our models both from a
quantitative and qualitative standpoint by conducting an intricate spatial
analysis of their performance in relation to the locations of M- and X-class
solar flares, that were used as the labels. For the purpose of our analysis,
we used the predictions made on the test set and created a heatmap by
gathering the flares grouped by their location in the Heliographic Stonyhurst
(HGS) coordinate system, where each bin represents a 5∘ $\times$ 5∘ spatial
cell in terms of latitude and longitude. Initially, we computed the recall for
the $\geq$M-class flares (combined M- and X-class flares) in each spatial
cell, providing a comprehensive assessment of the models’ performance.
Subsequently, we evaluated the recall separately for M-class and X-class
flares, allowing us to analyze the models’ sensitivity at a more granular
level. The heatmaps that illustrate the spatial distribution of recall scores
for $\geq$M-, X-, and M-class flares are shown in Fig. 4, 5 (a), and 5 (b)
respectively. This allowed us to compare all three models on their
capabilities to learn spatial patterns that localize the regions where the
models were more effective in making accurate predictions and vice versa.

Our findings indicate that all three models demonstrated reasonable
proficiency in predicting X-class flares in central locations. However, among
these, the ResNet34-based model stood out for its overall better performance
in accurately forecasting X-class flares, regardless of whether they were in
near-limb or central locations as shown in Fig. 5 (a). Upon analysis of the
heatmaps for $\geq$M- and only M-class flares, as depicted in Fig. 4 and Fig.
5 (b) respectively, we observed that the ResNet34-based model generally
yielded more accurate predictions across diverse spatial locations in
comparison to the other models. Nonetheless, a common limitation across all
three models was an elevated rate of false negatives in near-limb areas for
M-class flares. Notably, these regions are often associated with unreliable
readings due to projection effects. Despite this challenge, our study
signifies a substantial progression in solar flare prediction, enabling the
prediction of flares even in these intricate near-limb regions with distorted
magnetic fields. This ability to predict flares in near-limb areas has
considerable implications in operational forecasting.

##  V Conclusion and Future Work

Our study involved the evaluation of three deep learning models, namely
AlexNet, VGG16, and ResNet34, for the prediction of solar flares, with a
primary focus on their performance for near-limb events. Through rigorous
evaluation of our models and analysis of the prediction results, we observed a
notable performance advantage of the ResNet34-based model in predicting near-
limb flares. This finding highlights the efficacy of employing deeper
architectures with residual connections, which enhance feature extraction and
facilitate the capture of subtle patterns associated with near-limb events.
Moreover, our study highlighted the variability in model performance across
different flare types and event locations, emphasizing the importance of
tailoring models and analyzing results in context-specific manners. This shows
the need for further exploration of deep learning models to effectively
capture the diverse nature of flare events. The implications of our research
extend to operational forecasting systems, where the precise and reliable
prediction of solar flares, including near-limb events, holds significant
importance. Future research directions can explore the integration of multi-
modal data and the development and utilization of deep learning models that
can learn from temporally evolving solar activity.

Acknowledgements: This work is supported in part under two NSF grants (Award
#2104004 and #1931555) and a NASA SWR2O2R grant (Award #80NSSC22K0272).

## References

  * [1] L. Fletcher, B. R. Dennis, H. S. Hudson, S. Krucker, K. Phillips, A. Veronig, M. Battaglia, L. Bone, A. Caspi, Q. Chen, P. Gallagher, P. T. Grigis, H. Ji, W. Liu, R. O. Milligan, and M. Temmer, “An observational overview of solar flares,” _Space Science Reviews_ , vol. 159, no. 1-4, pp. 19–106, Aug. 2011.
  * [2] C. Pandey, A. Ji, R. A. Angryk, M. K. Georgoulis, and B. Aydin, “Towards coupling full-disk and active region-based flare prediction for operational space weather forecasting,” _Frontiers in Astronomy and Space Sciences_ , vol. 9, Aug. 2022. [Online]. Available: https://doi.org/10.3389/fspas.2022.897301
  * [3] C. Pandey, M. K. Georgoulis, B. Aydin, R. A. Angryk, and A. Ji, “Exploring heuristics in full-disk aggregation from individual active region prediction of solar flares,” in _44th COSPAR Scientific Assembly. Held 16-24 July_ , vol. 44, 2022, p. 3457.
  * [4] J. T. Hoeksema, Y. Liu, K. Hayashi, X. Sun, J. Schou, S. Couvidat, A. Norton, M. Bobra, R. Centeno, K. D. Leka, G. Barnes, and M. Turmon, “The helioseismic and magnetic imager (HMI) vector magnetic field pipeline: Overview and performance,” _Solar Physics_ , vol. 289, no. 9, pp. 3483–3530, Mar. 2014.
  * [5] D. A. Falconer, S. K. Tiwari, R. L. Moore, and I. Khazanov, “A new method to quantify and reduce the net projection error in whole-solar-active-region parameters measured from vector magnetograms,” _The Astrophysical Journal_ , vol. 833, no. 2, p. L31, Dec. 2016.
  * [6] X. Huang, H. Wang, L. Xu, J. Liu, R. Li, and X. Dai, “Deep learning based solar flare forecasting model. i. results for line-of-sight magnetograms,” _The Astrophysical Journal_ , vol. 856, no. 1, p. 7, Mar. 2018.
  * [7] A. Ji, B. Aydin, M. K. Georgoulis, and R. Angryk, “All-clear flare prediction using interval-based time series classifiers,” in _2020 IEEE International Conference on Big Data (Big Data)_. IEEE, Dec. 2020, pp. 4218–4225. [Online]. Available: https://doi.org/10.1109/bigdata50022.2020.9377906
  * [8] M. D. Crown, “Validation of the NOAA space weather prediction center's solar flare forecasting look-up table and forecaster-issued probabilities,” _Space Weather_ , vol. 10, no. 6, pp. n/a–n/a, Jun. 2012.
  * [9] K. Lee, Y.-J. Moon, J.-Y. Lee, K.-S. Lee, and H. Na, “Solar flare occurrence rate and probability in terms of the sunspot classification supplemented with sunspot area and its changes,” _Solar Physics_ , vol. 281, no. 2, pp. 639–650, Sep. 2012.
  * [10] K. Kusano, T. Iju, Y. Bamba, and S. Inoue, “A physics-based method that can predict imminent large solar flares,” _Science_ , vol. 369, no. 6503, pp. 587–591, Jul. 2020.
  * [11] X. Li, Y. Zheng, X. Wang, and L. Wang, “Predicting solar flares using a novel deep convolutional neural network,” _The Astrophysical Journal_ , vol. 891, no. 1, p. 10, Feb. 2020.
  * [12] K. Whitman, R. Egeland, I. G. Richardson, C. Allison, P. Quinn, J. Barzilla, I. Kitiashvili, V. Sadykov, H. M. Bain, M. Dierckxsens, M. L. Mays, T. Tadesse, K. T. Lee, E. Semones, J. G. Luhmann, M. Núñez, S. M. White, S. W. Kahler, A. G. Ling, D. F. Smart, M. A. Shea, V. Tenishev, S. F. Boubrahimi, B. Aydin, P. Martens, R. Angryk, M. S. Marsh, S. Dalla, N. Crosby, N. A. Schwadron, K. Kozarev, M. Gorby, M. A. Young, M. Laurenza, E. W. Cliver, T. Alberti, M. Stumpo, S. Benella, A. Papaioannou, A. Anastasiadis, I. Sandberg, M. K. Georgoulis, A. Ji, D. Kempton, C. Pandey, G. Li, J. Hu, G. P. Zank, E. Lavasa, G. Giannopoulos, D. Falconer, Y. Kadadi, I. Fernandes, M. A. Dayeh, A. Muñoz-Jaramillo, S. Chatterjee, K. D. Moreland, I. V. Sokolov, I. I. Roussev, A. Taktakishvili, F. Effenberger, T. Gombosi, Z. Huang, L. Zhao, N. Wijsen, A. Aran, S. Poedts, A. Kouloumvakos, M. Paassilta, R. Vainio, A. Belov, E. A. Eroshenko, M. A. Abunina, A. A. Abunin, C. C. Balch, O. Malandraki, M. Karavolos, B. Heber, J. Labrenz, P. Kühl, A. G. Kosovichev, V. Oria, G. M. Nita, E. Illarionov, P. M. O’Keefe, Y. Jiang, S. H. Fereira, A. Ali, E. Paouris, S. Aminalragia-Giamini, P. Jiggens, M. Jin, C. O. Lee, E. Palmerio, A. Bruno, S. Kasapis, X. Wang, Y. Chen, B. Sanahuja, D. Lario, C. Jacobs, D. T. Strauss, R. Steyn, J. van den Berg, B. Swalwell, C. Waterfall, M. Nedal, R. Miteva, M. Dechev, P. Zucca, A. Engell, B. Maze, H. Farmer, T. Kerber, B. Barnett, J. Loomis, N. Grey, B. J. Thompson, J. A. Linker, R. M. Caplan, C. Downs, T. Török, R. Lionello, V. Titov, M. Zhang, and P. Hosseinzadeh, “Review of solar energetic particle models,” _Advances in Space Research_ , Aug. 2022. [Online]. Available: https://doi.org/10.1016/j.asr.2022.08.006
  * [13] C. Pandey, R. A. Angryk, M. K. Georgoulis, and B. Aydin, “Explainable deep learning-based solar flare prediction with post hoc attention for operational forecasting,” 2023. [Online]. Available: https://arxiv.org/abs/2308.02682
  * [14] C. Pandey, A. Ji, T. Nandakumar, R. A. Angryk, and B. Aydin, “Exploring deep learning for full-disk solar flare prediction with empirical insights from guided grad-cam explanations,” 2023. [Online]. Available: https://arxiv.org/abs/2308.15712
  * [15] J. Hong, A. Ji, C. Pandey, and B. Aydin, “Beyond traditional flare forecasting: A data-driven labeling approach for high-fidelity predictions,” in _Big Data Analytics and Knowledge Discovery_. Springer Nature Switzerland, 2023, pp. 380–385. [Online]. Available: https://doi.org/10.1007/978-3-031-39831-5_34
  * [16] T. Aso, T. Ogawa, and M. Abe, “Application of back-propagation neural computing for the short-term prediction of solar flares.” _Journal of geomagnetism and geoelectricity_ , vol. 46, no. 8, pp. 663–668, 1994.
  * [17] M. G. Bobra, X. Sun, J. T. Hoeksema, M. Turmon, Y. Liu, K. Hayashi, G. Barnes, and K. D. Leka, “The helioseismic and magnetic imager (HMI) vector magnetic field pipeline: SHARPs – space-weather HMI active region patches,” _Solar Physics_ , vol. 289, no. 9, pp. 3549–3578, Apr. 2014.
  * [18] C. Pandey, R. A. Angryk, and B. Aydin, “Solar flare forecasting with deep neural networks using compressed full-disk HMI magnetograms,” in _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, Dec. 2021, pp. 1725–1730. [Online]. Available: https://doi.org/10.1109/bigdata52589.2021.9671322
  * [19] C. Pandey, R. Angryk, and B. Aydin, “Deep neural networks based solar flare prediction using compressed full-disk line-of-sight magnetograms,” in _Information Management and Big Data_. Springer International Publishing, 2022, pp. 380–396. [Online]. Available: https://doi.org/10.1007/978-3-031-04447-2_26
  * [20] C. Pandey, R. A. Angryk, and B. Aydin, “Explaining full-disk deep learning model for solar flare prediction using attribution methods,” 2023. [Online]. Available: https://arxiv.org/abs/2307.15878
  * [21] A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” 2014.
  * [22] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014.
  * [23] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  * [24] J. Schou, P. H. Scherrer, R. I. Bush, R. Wachter, S. Couvidat, M. C. Rabello-Soares, R. S. Bogart, J. T. Hoeksema, Y. Liu, T. L. Duvall, D. J. Akin, B. A. Allard, J. W. Miles, R. Rairden, R. A. Shine, T. D. Tarbell, A. M. Title, C. J. Wolfson, D. F. Elmore, A. A. Norton, and S. Tomczyk, “Design and ground calibration of the helioseismic and magnetic imager (HMI) instrument on the solar dynamics observatory (SDO),” _Solar Physics_ , vol. 275, no. 1-2, pp. 229–259, Oct. 2011.
  * [25] W. Pesnell, B. J. Thompson, and P. C. Chamberlin, “The solar dynamics observatory (SDO),” _Solar Physics_ , vol. 275, no. 1-2, pp. 3–15, Oct. 2011.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” 2015.
  * [27] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” 2017.
  * [28] DMLab, “Source code,” 2023. [Online]. Available: https://bitbucket.org/gsudmlab/fulldisk-spatial-analytics/src/main/


# Line as a Visual Sentence:
Context-aware Line Descriptor for Visual Localization

Sungho Yoon1 and Ayoung Kim2∗ Manuscript received: April 29, 2021; Revised
August 2, 2021; Accepted September 7, 2021. This paper was recommended for
publication by Editor Sven Behnke upon evaluation of the Associate Editor and
Reviewers’ comments. This work was fully supported by [Localization in
changing city] project funded by Naver Labs Corporation.1S. Yoon is with the
Robotics Program, KAIST, Daejeon, S. Korea<EMAIL_ADDRESS>Kim is
with the Department of Mechanical Engineering, SNU, Seoul, S. Korea
<EMAIL_ADDRESS>Object Identifier (DOI): see top of this page.

###### Abstract

Along with feature points for image matching, line features provide additional
constraints to solve visual geometric problems in robotics and CV (CV).
Although recent CNN (CNN)-based line descriptors are promising for viewpoint
changes or dynamic environments, we claim that the CNN architecture has innate
disadvantages to abstract variable line length into the fixed-dimensional
descriptor. In this paper, we effectively introduce Line-Transformers dealing
with variable lines. Inspired by NLP (NLP) tasks where sentences can be
understood and abstracted well in neural nets, we view a line segment as a
sentence that contains points (words). By attending to well-describable points
on a line dynamically, our descriptor performs excellently on variable line
length. We also propose line signature networks sharing the line’s geometric
attributes to neighborhoods. Performing as group descriptors, the networks
enhance line descriptors by understanding lines’ relative geometries. Finally,
we present the proposed line descriptor and matching in a PL-Loc (PL-Loc). We
show that the visual localization with feature points can be improved using
our line features. We validate the proposed method for homography estimation
and visual localization.

###### Index Terms:

Localization, SLAM, Deep Learning for Visual Perception.

##  I Introduction

Visual features are widely utilized in many robotics and CV (CV) applications
such as visual tracking, SLAM (SLAM), and SFM (SFM). While the keypoint
features are well-studied and a base of many applications, keypoints that are
not well distributed in the image may result in unstable and inaccurate pose
estimation.

Recent research has reported that SLAM performance can be enhanced by using
both points and lines [1, 2, 3, 4] even for low-textured environments. For
example, LBD (LBD) [5] is one of the widely-used line descriptors in SLAM.
Reliable performance of LBD has been reported for continuous frames, whereas
the performance degrades under a wide baseline, which prevented the line-based
approach from adapting line features directly in visual localization [6, 7].
Tackling this limitation, approaches leveraged CNN (CNN) to learn the
representation of line descriptors [8, 9, 10] yielding superior performance.
Still, CNN has innate difficulties in dealing with variable line lengths
because the dimension of CNN should be predefined by its architecture. Another
issue of line occlusion is also related to this length variation and was
examined in SOLD2 [11].

Figure 1: (a) For both low and high-textured environments, our line descriptor
performs accurate matching. (b) Attention scores for points in a line with
matched lines having similar attentional contexts.

In our work, we overcome this fixed length requirement and achieve flexibility
to the length variation by interpreting a line segment from the NLP (NLP)
point of view. We view a line consisting of points as a sentence consisting of
words, thereby allowing various lengths for lines. The word2vec in NLP learns
vector representations that encapsulate each distinct word, leveraging them as
base inputs of RNN, LSTM, or transformers for later tasks such as text
classification and text summarization. Similarly, we utilize a learned
descriptor map as a transition from RGB pixels to point vectors.

Based on this key idea, we propose a novel line descriptor using transformers
[12]. The transformers in our model learn the context of the point vectors
within a line segment to summarize it in the form of a line descriptor. Unlike
SOLD2, a line is not just a simple sequence of points but is handled with
attention to its context dynamically, enabling reliable performance even under
occlusion. The attributes of the proposed method can be summarized below.

Figure 2: Line-Transformers consist of two major components: line transformers
and line signature networks. The first component uses a line tokenizer to
extract point tokens and embeddings from a line segment. Considering the
context of the point embeddings, transformers summarize it into a line
embedding, or a line descriptor. The second component enhances the line
descriptor by sharing lines’ positional context with its neighborhoods.

  * •

We present a novel line segment descriptor using a transformer architecture by
considering line segments as sentences and points as words. Leveraging NLP for
line descriptor, we can handle lines with various lengths.

  * •

The proposed line descriptor understands the line segment’s context by paying
attention to more meaningful points on a line. It effectively abstracts
various length lines to a fixed-sized descriptor.

  * •

We suggest line signature networks that share the message of line attributes
(e.g., position, angle, length, and descriptors) between neighborhoods.
Serving as a grouped line descriptor, the line descriptors can learn geometric
attributes of their neighborhoods.

##  II Related Work

###  II-A Line Descriptors

As a handcrafted line descriptor, Wang et al. [13] proposed a MSLD (MSLD)
observing gradients around line segments. LBD [5] enhanced the precision and
computing time by investigating gradients on bands that are parallel to a
line. Recently, Vakhitov and Lempitsky [8] presented a LLD (LLD) for visual
SLAM in a deep-learning manner. They first built a full-sized descriptor map
using CNN and uniformly split a line segment into a fixed-number of
subsegments. Lange et al. proposed a learning-based line descriptor named DLD
[9]. They also split a line segment and trained the CNN with a self-generated
dataset. Extended works from the same group [10] adapted wavelets to extract
more meaningful information from an image, referred to as the WLD (WLD).
Pautrat et al. [11] introduced SOLD2 for joint line detection and description.
To find a line correspondence, SOLD2 sampled uniformly-spaced point
descriptors on a line and performed the sequence matching.

###  II-B Point Descriptors

The recent point descriptor also focused on learning-based methods. Like a
handcrafted feature SIFT [14], learned descriptors often utilized the patch as
their inputs as in L2-Net [15] and HardNet [16]. Examining keypoint detection
and description together, LIFT [17] introduced an end-to-end learning pipeline
implementing keypoint detection, orientation estimation, and description
simultaneously. SuperPoint [18] proposed self-supervised learning for detector
and descriptor using synthetic pretraining and homography adaptation.

###  II-C Point-and-Line SLAM

Exploiting both point and line for SLAM has strong advantages, robustly
performing in challenging environments such as low-textured, motion blurred,
and defocused cases [1, 2, 3, 4]. PL-SVO [1] extended the SVO method by
adapting its semi-direct approach to line segments. PL-SLAM [2] presented line
features on the monocular ORB-SLAM. In [3], stereo camera-based PL-SLAM was
presented to improve the bag-of-words method using points and lines.

In this paper, we utilize visual localization to evaluate line descriptors.
Visual localization [6, 7] is a problem of estimating a camera pose given a
query image and a prior 3D feature map, which is similar to relocalization in
SLAM. It differs from the previous works in considering only discrete scenes,
including large viewpoint changes, and it is effective for evaluating the
robustness of line descriptors in various situations.

###  II-D Transformers in NLP and CV

We briefly summarize recent studies connecting NLP and CV. Vaswani et al.
proposed the transformer for language translation [12], and it became a base
architecture of many state-of-the-art methods in NLP tasks. BERT (BERT) is one
of the widely used models utilizing transformer encoders [19]. One of the
recent trends in CV is the adoption of the transformers [20, 21]. Dosovitskiy
et al. proposed a Vision Transformer (ViT) [20] in which 16x16 patches from an
image are used as inputs of a standard transformer for image classification.
Visual Transformer (VT) [21] adopted a transformer using semantic tokens from
convolutions and spatial attention so as to treat an image focussing more on
important regions.

##  III Method

The proposed Line-Transformers aim to build a line descriptor given points
located on a line segment (Fig. 2).

Figure 3: (a) In BERT [19], a sentence is tokenized, and a [CLS] is prepended
for a sentence classification task. (b) Similar to BERT, a line segment is
tokenized, and a special line token [LINE] is used for aggregating points
information. (c) The point embedding is a descriptor vector located on each
point token on a descriptor map.

###  III-A Line Transformers

####  III-A1 Line Tokenizer

We briefly illustrate NLP terminologies and explain our method based on a
similar concept. In NLP, word tokenization is a process used to divide a
sentence into smaller words called tokens. The tokens are utilized as minimum
units for model inputs. Special tokens are also used to perform special tasks.
For example, the classification token [CLS] is applied to aggregate sequence
representation for classification tasks and the separator token [SEP] is used
to differentiate two sentences. These tokens are converted into vector
representations by allocating words with similar meaning into a similar vector
space. The vector representation of tokens is called word embedding, and NLP
models exploit them to understand natural languages effectively. In our
problem, we formulate a point-and-line segment as the relationship between a
word and a sentence in the natural language.

As in Fig. 3, the line tokenizer aims to generate point tokens and their
embeddings that describe a line segment. After detecting line segments from an
image [22], we uniformly select points $\boldsymbol{\mathrm{p}}$ on a line
segment. The position of point $\boldsymbol{\mathrm{p}}$ consists of its image
coordinates and keypoint confidence:
$\boldsymbol{\mathrm{p_{i}}}=(x,y,c)_{i}$. The inter-points interval ($v$) is
then determined depending on the level of discriminability. When the interval
of points is small, the model can receive more information to describe a line
but needs larger computations and memory. The number of points ($n$) on a line
is $n=\lfloor\ell/v\rfloor+1$ when a line length is $\ell$. As the [CLS] token
in BERT [19], we introduced a special line token [LINE] to aggregate
contextual information of point tokens. The line token is prepended on the
series of point tokens.

We encode each point token into a vector representation, point embedding
$\boldsymbol{\mathrm{E}}\in\mathbb{R}^{1\times D}$, where $D$ is the
descriptor dimension. This is done by associating each point token with a
vector that matched in a dense descriptor map [18] encoded with a descriptor
vector at each pixel. The embedding of the special token [LINE],
$\boldsymbol{\mathrm{E_{line}}}\in\mathbb{R}^{1\times D}$, expresses the
initial state of a line descriptor, and its weight values are learned during
the training process. Finally, we add the embeddings to the positional
embeddings $\boldsymbol{\mathrm{E_{pos}\in\mathbb{R}^{(n+1)\times D}}}$ which
is obtained by MLP (MLP) using each point’s position.

####  III-A2 Transformer

Given the token embeddings, we model a line descriptor using transformers
[12]. The transformer encoder is composed of two sublayers: MSA (MSA) layers
and MLP layers, whereas each sub-layer has a residual connection and LN (LN).

We stack the transformer $L$ times as in (1). Here, the token embeddings
$\boldsymbol{\mathrm{z_{0}}}$ serve as the encoder inputs. The line embedding
$\boldsymbol{\mathrm{E_{line}}}$ is located at the first element of
$\boldsymbol{\mathrm{z_{0}}}$, and is expressed with the superscript $0$.
Then, at the last layer $\boldsymbol{\mathrm{z_{\textit{L}}}}$, the line
embedding contains the line context (i.e.,
$\boldsymbol{\mathrm{E_{line}=\boldsymbol{\mathrm{z_{0}^{0}}}}}$ and a line
descriptor
$\boldsymbol{\mathrm{d}}=\boldsymbol{\mathrm{z_{\textit{L}}^{0}}}$).

$\displaystyle\boldsymbol{\mathrm{z_{0}}}$ $\displaystyle=$
$\displaystyle[\boldsymbol{\mathrm{E_{line}}};\boldsymbol{\mathrm{E_{1}}};\boldsymbol{\mathrm{E_{2}}};...;\boldsymbol{\mathrm{E_{\textit{n}}}}]+\boldsymbol{\mathrm{E_{pos}}},$
(1) $\displaystyle\boldsymbol{\mathrm{z^{\prime}_{\textit{i}-1}}}$
$\displaystyle=$
$\displaystyle\text{LN}(\text{MSA}(\boldsymbol{\mathrm{z_{\textit{i}-1}}},\boldsymbol{\mathrm{mask_{0}}})+\boldsymbol{\mathrm{z_{\textit{i}-1}}}),$
$\displaystyle\boldsymbol{\mathrm{z_{\textit{i}}}}$ $\displaystyle=$
$\displaystyle\text{LN}(\text{MLP}(\boldsymbol{\mathrm{z^{\prime}_{\textit{i}-1}}})+\boldsymbol{\mathrm{z^{\prime}_{\textit{i}-1}}}),\textit{i}=1...\textit{L}$
$\displaystyle\boldsymbol{\mathrm{d}}$ $\displaystyle=$
$\displaystyle\boldsymbol{\mathrm{z_{\textit{L}}^{0}}}$

Each line segment has a various number of point tokens based on the length
$\ell$. To handle the various sizes at once, we use a mask vector
$\boldsymbol{\mathrm{mask_{0}}}$ for the MSA to reject the unrelated point
embeddings inside of scaled dot-product attention. The mask has the size
$1\times n_{max}$.

###  III-B Line Signature Networks

Inspired by line signatures [23] and message-passing [24, 25] in GNN (GNN),
the proposed deep line signature networks are designed for line signatures’
messages to be shared with each line segment using graph attention networks.
The line signature is originally proposed as a grouped line descriptor. It
clusters neighbor line segments as a group and takes relative positions by a
series of angles and length ratios. While the line signature needs to define
the clustering range of neighbor line segments and the man-defined attributes
of lines, we exploit a graph attention network that can implicitly assign
neighbor line segments and pass the messages of their descriptions including
positions.

$\displaystyle\boldsymbol{\mathrm{d^{\prime}_{\textit{i}}}}$ $\displaystyle=$
$\displaystyle\boldsymbol{\mathrm{d_{\textit{i}}}}+\text{MLP}(\boldsymbol{\mathrm{x_{\textit{i}}}},\boldsymbol{\mathrm{y_{\textit{i}}}},\ell_{\textit{i}},\cos\theta_{\textit{i}},\sin\theta_{\textit{i}})$
(2) $\displaystyle\boldsymbol{\mathrm{s_{0}}}$ $\displaystyle=$
$\displaystyle[\boldsymbol{\mathrm{d^{\prime}_{1}}};\boldsymbol{\mathrm{d^{\prime}_{2}}};...;\boldsymbol{\mathrm{d^{\prime}_{\textit{m}}}}]$
$\displaystyle\boldsymbol{\mathrm{s_{\textit{j}}}}$ $\displaystyle=$
$\displaystyle\boldsymbol{\mathrm{s_{\textit{j}-1}}}+\text{MLP}\Big{(}\boldsymbol{\mathrm{s_{\textit{j}-1}}}||\text{MSA}(\boldsymbol{\mathrm{s_{\textit{j}-1}}})\Big{)},\textit{j}=1...\textit{M}$

We first make an attribute embedding by feeding the line’s attributes such as
midpoints $(x,y)$, angle $\theta$, and length $\ell$ to MLP. Then, we add it
on descriptor $\boldsymbol{\mathrm{d_{i}}}$ and share the messages with each
line segment using the message-passing network as in (2). The operator $||$
represents concatenation, and $m$ denotes the number of line descriptors
within an image. We also stack the message-passing layers $M$ times. Finally,
we normalize line descriptors in $\boldsymbol{\mathrm{z_{\textit{M}}}}$ after
feeding them to another MLP.

Figure 4: Keylines, sublines, and adjacency matrix. Points are selected with
the interval $v$ on a line segment. When the number of points $n$ is greater
than $n_{max}$, the keyline is divided into two sublines as in (b). The
adjacency matrix describes relationship between keylines and sublines.

###  III-C Sublines to Keylines

For transformers, the number of input tokens is limited by its maximum size
($n_{max}$). Overcoming the limited text length of transformers, the longer
sequences were often truncated in NLP. Instead of truncation, we handle long
line segments utilizing the keyline and subline concepts. Let us call the
original line segment a keyline. When the keyline is longer than the maximum
($\ell>n_{max}\times v$), we divide it into multiple sublines. In doing so, we
can build an adjacency matrix $\mathrm{A_{img}}$ via the relationship between
keylines and sublines. The values of the adjacency matrix are one divided by
the number of sublines, as presented in Fig. 4(b). Then, the distance matrix
of sublines $\mathrm{C_{sublines}}$ can be transformed into a distance matrix
of keylines $\mathrm{C_{keylines}}$ as follows:

$\mathbf{\mathrm{C_{keylines}}=\mathrm{A_{img1}}\cdot\mathrm{C_{sublines}}\cdot\mathrm{A_{img2}}^{\top}},$

where the adjacency matrices of two images are $\mathrm{A_{img1}}$ and
$\mathrm{A_{img2}}$. The distance matrix of sublines includes the distance
between descriptors from two images, which can be calculated by a matcher such
as nearest neighbor.

Similar to the ensemble averaging, (III-C) averages the multiple sublines’
distances with respect to a keyline via matrix multiplication of adjacency
matrix with distance matrix. For example, the distance between keylines in
image1 and sublines in image2 can be represented in
$\mathrm{A_{img1}}\cdot\mathrm{C_{sublines}}$.

###  III-D Loss Function

We use a triplet loss function with semi-hard negative sampling strategy [26].
The basic idea of triplet loss is to make the distance between an anchor
descriptor $\boldsymbol{\mathrm{a_{i}}}$ and its matched (positive) descriptor
$\boldsymbol{\mathrm{p_{i}}}$ closer and to simultaneously further the
distance with a nonmatched (negative) descriptor
$\boldsymbol{\mathrm{n_{i}}}$. In line segment matching, one line in one image
can be matched with more than two lines in another image and this means that a
single anchor line can have multiple positive lines. In our implementation, we
choose the most overlapped line as the positive $\boldsymbol{\mathrm{p_{i}}}$.
The overall loss function is given as:

$\mathcal{L}=\frac{1}{n}\sum_{\textit{i}}^{n}\max(0,\alpha+\left\lVert{\boldsymbol{\mathrm{a_{\textit{i}}}}-\boldsymbol{\mathrm{p_{\textit{i}}}}}\right\rVert^{2}-\left\lVert{\boldsymbol{\mathrm{a_{\textit{i}}}}-\boldsymbol{\mathrm{n^{\prime}_{\textit{i}}}}}\right\rVert^{2}),$
(3)

where the semi-hard negative sample
$\boldsymbol{\mathrm{n^{\prime}_{\textit{i}}}}$ is chosen to the hard negative
further away from the positive $\boldsymbol{\mathrm{p_{\textit{i}}}}$. We
observed that semi-negative sampling helped converging loss values stably. The
margin value $\alpha$ provides the capacity to increase the negative
distances, and we set it to $1$.

###  III-E Implementation Details

To detect line segments on images, we used LSD (LSD) [22] for its high
generalizability over various environments. We selected SuperPoints [18] for
our front-end descriptor map. Because its raw descriptor map is sized
$H/8\times W/8$ given an image sized $H\times W$, we set 8 to the interval of
points on a line. We limited the number of point tokens greater than 2 and
smaller than 21 for a subline. A line descriptor, key, query, and value in MSA
have the same dimension $\mathbf{D}=256$ as the SuperPoint. The MSA has four
heads, and the line transformers and line signature networks have $L=12$ and
$M=7$ layers, respectively. Our networks contain 14M parameters, and they run
at an average speed of 20 ms on an NVIDIA GTX 2080Ti GPU for 256 line
descriptors in an image. It is implemented in Pytorch using the Adam optimizer
with a learning rate of 0.001.

Figure 5: Qualitative line segment matches for homography estimation and
visual localization. The last three columns represent blurring, viewpoint, and
illumination changes. More correct matches (green) and fewer wrong matches
(red) indicate better performance. Unmatched lines are colored in blue.

##  IV Experiments

We evaluate our line descriptor in terms of homography estimation and visual
localization performance. For two test scenarios, we compare the proposed
method against SuperPoint, handcrafted line descriptor LBD [5], learning-based
line descriptors LLD [8], WLD [10], and SOLD2 [11]. We use a NN (NN) matcher
for LBD, LLD, and WLD, and SOLD2 by means of its own line matcher. We include
SuperPoint as a reference for the point-based matching. We used a maximum of
512 and 1,024 points at each dataset. For lines, we extracted 256 line
segments in the longer order. Further results can be seen in the video line-
as-a-visual-sentence.mp4.

###  IV-A Homography Estimation

####  IV-A1 Datasets

For homography estimation, we use Oxford and Paris dataset [27]. The dataset
includes 5K (Oxford) and 6K (Paris) images and we split them into the
training, validatation, and test sets. For self-supervised learning, we
augment images using synthetic homographies and photometric distortions [18,
24, 28].

To establish ground-truth line correspondences from an image pair, we first
detect line segments from both the original image and its augmented image.
Then, we project two endpoints of each line onto one another using a known
homography matrix. The criteria for the true correspondence are (i) overlap
existence, (ii) reprojection error ($<$ $4\text{\,}\mathrm{p}\mathrm{x}$), and
(iii) angle difference ($<2^{\circ}$). The resulting true correspondences were
expressed as an overlap-similarity matrix. The overlap similarity between two
lines measures the ratio between overlapped line length and smaller line
length as $O(L_{1},L_{2})=L_{1}\cap L_{2}/\min(L_{1},L_{2})$. The overlapped
line length $L_{1}\cap L_{2}$ is the distance between two middle endpoints
among four endpoints of two lines. For SuperPoint, true point correspondences
are defined by the point-wise reprojection errors ($<$
$4\text{\,}\mathrm{p}\mathrm{x}$) following [24].

####  IV-A2 Metrics

We implement a line-based homography matrix estimation [29]. With the
homography estimation utilizing a RANSAC (RANSAC) of 2,000 iterations, we
compute average reprojection errors of the four image corners and report the
area under the cumulative error curve (AUC) at the thresholds (5, 10, and, 20
pixels). We also compute matching precision (P) and recall (R) based on the
ground-truth matches.

TABLE I: Homography estimation evaluation. The best-performing values are in
bold font.

| Homography AUC | P | R | F
---|---|---|---|---
AUC@5px | AUC@10px | AUC@20px
(SuperPoint) | (38.5) | (43.3) | (46.3) | (37.6) | (38.6) | (39.6)
LBD | 2.2 | 7.6 | 17.5 | 20.6 | 55.2 | 30.1
LLD | 0.7 | 2.6 | 6.6 | 5.9 | 13.6 | 8.3
WLD | 16.7 | 35.2 | 54.5 | 48.0 | 51.3 | 49.6
SOLD2 | 31.8 | 51.5 | 67.1 | 41.1 | 45.8 | 43.3
LT (Ours) | 29.5 | 52.1 | 69.4 | 57.7 | 61.5 | 59.5

####  IV-A3 Results

Table. I lists the quantitative comparison. Our Line-Transformers outperform
other line descriptor methods in terms of F-score by a large margin (10.1%).
Our method also outperforms every homography estimation metric except AUC
under five pixels. Compared to SuperPoint, the Line-Transformers yielded more
stable performance at the AUC under 10 and 20 pixels. LLD has low performances
on this dataset because it originally trained in continuous frames without
large viewpoint changes.

The precision and recall are a direct and explicit measure for line-matching
performance depending solely on the number of correct/wrong matches. More
implicit performance could be captured from homography estimation when the
performance depends on the number, distribution, and quality of the matches.
In that sense, the proposed method fulfilled the quantity and quality of the
reliable matches. We discuss the matching evaluation in more detail in §IV-D.

Fig. 5 shows qualitative results for the homography estimation-based line
matching. Line-Transformers have better performance by producing more correct
matches and fewer false matches than other line descriptors. It also shows
that LBD in particular has many incorrect matches, resulting in lower matching
precision.

(a) Performance difference between Line-Transformers and other descriptors.

|  | Precision | Recall | F-score
---|---|---|---|---
|  | Short | Mid | Long | Short | Mid | Long | Short | Mid | Long
Homography etimation | LBD | 10.9 | 20.8 | 28.0 | 40.5 | 54.9 | 55.0 | 17.1 | 30.2 | 37.1
LLD | 3.3 | 6.0 | 8.1 | 10.3 | 13.3 | 14.1 | 5.0 | 8.3 | 10.3
WLD | 24.9 | 43.5 | 53.9 | 43.3 | 55.4 | 55.0 | 31.6 | 48.8 | 54.4
SOLD2 | 24.3 | 39.6 | 50.0 | 29.5 | 48.5 | 47.7 | 26.6 | 43.6 | 48.8
LT (Ours) | 35.3 | 57.5 | 67.5 | 44.1 | 61.3 | 62.7 | 39.2 | 59.3 | 65.0
Visual localization | LBD | 11.3 | 17.7 | 27.1 | 42.3 | 47.9 | 53.4 | 17.8 | 25.9 | 35.9
LLD | 6.8 | 9.8 | 16.5 | 26.3 | 29.1 | 35.9 | 10.2 | 14.7 | 22.6
WLD | 18.6 | 25.6 | 38.7 | 33.8 | 35.3 | 39.9 | 24.0 | 29.7 | 39.3
SOLD2 | 24.7 | 30.8 | 45.7 | 30.8 | 40.4 | 44.4 | 27.4 | 35.0 | 45.0
LT (Ours) | 34.7 | 45.1 | 59.5 | 61.6 | 67.6 | 73.3 | 44.4 | 54.1 | 65.7

(b) Matching performances by line length.

Figure 6: Performance difference by line length. The figure (a) illustrates
that the overall graph has an upward trajectory, thus showing that our method
performs better than other CNN-based line descriptors when line segments
extend.

###  IV-B Visual Localization

Next, we evaluate line descriptors by estimating a camera pose in 3D line
maps. The evaluations were performed using the ScanNet [30] and Oxford Radar
RobotCar [31] datasets for indoor and outdoor experiments. For indoor
environments, we trained our networks in supervised learning, and initialized
them with the homography estimation’s weights. Then, the weights trained using
ScanNet were applied directly to outdoor localization to observe our networks’
generalizability.

####  IV-B1 Indoor

The ScanNet dataset [30] includes 2.5M views of RGB-D data and 3D camera poses
in 1,513 indoor scenes. We generate ground-truth line correspondences and 3D
line maps. Similarly, as in the research [32, 24], we selected sufficiently
overlapping image pairs (40–80%) by investigating the depth maps. Due to the
potential inaccuracy in the depth maps, we decomposed a line into point arrays
and examined the intermediate points to validate the line correspondences.
Then, we randomly select 350 image pairs from each of the four overlapped
scenes for the test set, a total of 1,400 image pairs. Because depth maps are
not sufficiently accurate to find all ground-truth line pairs, some correct
line correspondences may be counted as a false negative. Therefore, we
mitigate this issue by checking scene depth quality, as well as observing the
ratio between the numbers of all extracted line segments and the validated
line segments during projection/unprojection procedures.

####  IV-B2 Outdoor

The Oxford Radar RobotCar dataset [31] is built upon the Oxford RobotCar
dataset [33]. We selected two sequences (2019-01-11-12-26-55 and
2019-01-16-13-09-37) for our reference and query sets. Then, we randomly
selected 300 query images in a sequence and performed visual place recognition
[34] to identify corresponding reference images that have a 3D line feature
map. Instead of using the GPS (GPS) potential unreliable signals, we have
computed the ground-truth relative pose between query and reference images
using their point clouds via ICP (ICP). In the final evaluation sets, we
excluded query-reference image pairs with poor ICP fitness. The 3D line maps
were generated following procedures similar to those for our indoor
evaluation. Unlike the indoor RGB-D camera, however, the projected LiDAR
points are so sparse that 2D line segments can be difficult to find their
corresponding depth value. To alleviate this, we utilized a depth completion
[35].

TABLE II: Visual localization

| Visual Localization AUC (Indoor) | P | R | F
---|---|---|---|---
| 0.25m/10º | 0.50m/10º | 1.00m/10º
(SuperPoint) | (83.6 PnP) | (86.4 PnP) | (86.8 PnP) | (49.5) | (69.1) | (57.7)
| PnL | PnPL | PnL | PnPL | PnL | PnPL |  |  |
LBD | 15.7 | 38.4 | 19.0 | 45.9 | 19.3 | 46.6 | 18.5 | 49.8 | 27.0
LLD | 12.0 | 35.8 | 16.1 | 41.6 | 16.4 | 42.5 | 10.5 | 31.8 | 15.8
WLD | 28.0 | 55.9 | 35.4 | 61.6 | 36.1 | 62.4 | 27.9 | 37.3 | 31.9
SOLD2 | 46.4 | 73.5 | 57.4 | 77.9 | 59.2 | 78.6 | 35.0 | 40.5 | 37.5
LT (w/o LS) | 47.4 | 76.9 | 59.8 | 81.6 | 61.4 | 82.5 | 42.4 | 62.2 | 50.4
LT (Ours) | 53.0 | 79.2 | 65.0 | 83.3 | 66.6 | 83.9 | 49.4 | 68.4 | 57.3
| Visual Localization AUC (Outdoor) | P | R | F
| 0.25m/2º | 0.50m/5º | 5.00m/10º
(SuperPoint) | (37.1 PnP) | (63.6 PnP) | (90.8 PnP) |  |  |
| PnL | PnPL | PnL | PnPL | PnL | PnPL |  |  |
LBD | 1.9 | 4.5 | 2.6 | 18.5 | 9.1 | 59.2 | - | - | -
LLD | 1.9 | 7.5 | 6.4 | 30.2 | 20.0 | 71.7 | - | - | -
WLD | 9.4 | 21.5 | 29.4 | 45.7 | 54.7 | 83.4 | - | - | -
SOLD2 | 21.5 | 28.7 | 54.0 | 53.2 | 82.3 | 90.2 | - | - | -
LT (w/o LS) | 21.9 | 29.4 | 55.1 | 58.9 | 87.9 | 91.7 | - | - | -
LT (Ours) | 26.8 | 30.9 | 57.7 | 61.1 | 90.2 | 91.3 | - | - | -

####  IV-B3 Metrics

We report the percentage of correctly localized query images while using
different thresholds (i.e., 0.25m, 10°/ 0.5m, 10°/ 1.0m, 10° for indoor and
0.25m, 2°/ 0.5m, 5°/ 5.0m, 10° for outdoor). We estimated the camera pose by
PnPL (PnPL) [36] with a RANSAC of 20 iterations. For SuperPoint, we leveraged
PnP (PnP). We analyze the pose estimation results that use points, lines, and
points-and-lines, respectively. We also report matching precision (P) and
recall (R) based on the ground-truth matches at indoor.

####  IV-B4 Results

For both indoor and outdoor tests, Line-Transformers achieve the highest
performance among other line descriptors in visual localization and precision-
recall metrics (Table. II). The qualitative results in Fig. 5 illustrate that
the Line-Transformers perform robustly in imaging changes such as blurring,
viewpoints, and illuminations. Our method can be generalized well performing
reliably using the same weights trained in ScanNet datasets.

Unlike in the homography estimation, the point-based method with PnP
outperforms all line-based methods. One of the reasons is the small number of
3D line inliers during depth validation. While the 3D feature point is
directly determined by its corresponding depth pixel, some 3D line features
are filtered out during depth linearity’s validation in RANSAC. Thus, within
our mapping method, line-based localization is prone to performance
degradation than point-based. However, as will be discussed in §IV.F, the line
features can complement the performance of the points, especially when point
feature numbers are small or biased.

###  IV-C Performance by Line Length

To examine the robustness of our model with respect to various line lengths,
we further investigate performance comparisons against CNN-based methods while
varying line length. From each dataset, we evenly divide line segment sets
into three groups (i.e., short, mid, and long) by their lengths. Each group
has 33% numbers of all line segments, and the mid-length group ranged from
about 30 to 50 pixels.

As presented in Fig. 6(b), the proposed Line-Transformers outperform other
descriptors for all line lengths. Furthermore, as shown in Fig. 6(a), the
matching performance difference between Line-Transformers and other CNN-based
line descriptors (LLD and WLD) is increased by line length. This tendency
indicates that the performance of the proposed method is preserved even with
longer line segments, unlike the handcrafted descriptor and CNN-based
descriptor.

(a) Point-and-line features

Indoor Visual Localization
---
| #inliers | 0.25m/10º | 0.50m/10º | 1.00m/10º
SuperPoint | - | 83.6 | 86.4 | 86.8
PL-Loc | <20 | 83.6 | 86.5 | 87.2
<5 | 84.5 | 87.4 | 88.0
Best Feature (%) | P/L/PL | 61.0 / 6.3 / 32.7

(b) Point-and-Line Visual localization

Figure 7: Points and lines are complementary for better localization,
especially when keypoints are biased or in small numbers. TABLE III: Ablation
of Line-Transformers.

Line-Transformers | Line-based Visual Localization | P | R | F
---|---|---|---|---
0.25m/10º | 0.50m/10º | 1.00m/10º
No Line Signature Net (LS) | 47.4 | 59.8 | 61.4 | 42.4 | 62.2 | 50.4
No Positional encoding in LS | 49.4 | 62.8 | 64.3 | 44.3 | 66.4 | 53.1
No mid-point in LS | 51.1 | 62.6 | 64.9 | 47.7 | 69.0 | 56.4
No length, angle in LS | 51.9 | 64.2 | 65.8 | 44.0 | 68.4 | 53.6
Full | 53.0 | 65.0 | 66.6 | 49.4 | 68.4 | 57.3

###  IV-D Discussion on Evaluation Metrics

Unlike point features that assume one-to-one matching, line matching is a
many-to-many problem because a line detector tends to break the same line
segment into small lines differently at each image pair. For example, two non-
overlapping lines may originate from a single line due to the occlusion and
segmentation; thus, they should be considered the correct correspondences
semantically.

Evaluation of line segment matches often depends on human inspections [37, 10,
9, 5], assuming that line matches are one-to-one correspondences [5], not
many-to-many. Therefore, when a matcher finds only the closest matching pair,
the precision and recall should be carefully considered cautioning many-to-
many correspondences. In that sense, the precision-recall metrics may be
limited because they cannot consider non-overlapping line correspondences.

Visual localization and homography estimation could be more proper metrics in
this aspect. In visual localization, matching with the non-overlapped but
semantically same line is also considered a good match because a PnL (PnL)
algorithm does not consider the endpoint’s positions to ensure the robustness
for changing endpoints. Similarly, the line-based homography estimation does
not consider endpoints [29] but is limited to planar scenes in the real 3D
world. Hence, we found that line-based visual localization is a better
alternative that can inspect both overlapped and non-overlapped line matches
with large perspective changes.

###  IV-E Understanding Line-Transformers

To examine the attention scores of Line-Transformers, we visualize the maximum
attention score of each multi-heads within a line segment (Fig. 8). The point
embeddings on a line are not considered equally for a line descriptor, but
they have their own attention patterns. We also observe that the matched lines
have similar attention patterns and the line descriptors tend to refer to the
endpoints of a line.

The Fig. 8 (b) illustrates the attention between line descriptors in line
signature networks. We observe that the attentions gradually focus on a small
number of neighbor lines at a later layer. To study the quantitative effect of
the line signature networks, we evaluate our models without line signature
networks. As presented in Table. II and Table. III, the line signature
networks improve the localization results from 47.4% to 53.0% under the
threshold of 0.25m and 10°.

Figure 8: Visualizing attention scores. (a) Attention patterns in lines
describe how much the points embeddings contribute to building line
descriptors. The matched lines follow similar attention patterns. (b)
Attention scores between line descriptors are initially low and widely spread,
and they are gradually converged onto a small number of neighbor lines at a
later layer.

###  IV-F Visual Localization Using Feature Points and Lines

Despite the generally better performance in Table. II, the point-based
localization may be deteriorated due to the small number of points (e.g.,
feature-less environment) or biased feature distribution as in the sample
cases in Fig. 7. This section examines how the line-based approach can enhance
point-based localization in a complementary fashion.

For the analysis, we define point-based localization failure using the
reprojection errors of 3D features and count the inlier when the reprojection
error is less than four pixels. Then, the PL-Loc was additionally performed
for the cases when the number of inliers is less than 5 or 20 (i.e., when
point-based inliers drop).

As presented in the table in Fig. 7(b), the PL-Loc provides additional
enhancements to visual localization. More interestingly, the point
outperformed over the line for 61% of cases, which indicates the remaining 39%
of cases are with potential to be improved with lines. This also implies the
proper combination of point and line would improve the overall localization
performance. For example, better metrics followed by a strong model selection
could be examined to complete robust PnPL in future work.

##  V Conclusion

This paper presented a novel line descriptor handling variable line length
effectively using attention mechanisms, inspired by NLP tasks handling
sentences and paragraphs of various lengths. We also presented a PL-Loc
pipeline leveraging keypoints and keylines simultaneously for visual
localization. Our experiments demonstrated that our line descriptor achieved
state-of-the-art performance in homography estimation and visual localization
datasets.

## References

  * Gomez-Ojeda et al. [2016] R. Gomez-Ojeda, J. Briales, and J. Gonzalez-Jimenez, “Pl-svo: Semi-direct monocular visual odometry by combining points and line segments,” in _Proc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys._ , 2016, pp. 4211–4216.
  * Pumarola et al. [2017] A. Pumarola, A. Vakhitov, A. Agudo, A. Sanfeliu, and F. Moreno-Noguer, “Pl-slam: Real-time monocular visual slam with points and lines,” in _Proc. IEEE Intl. Conf. on Robot. and Automat._ , 2017, pp. 4503–4508.
  * Gomez-Ojeda et al. [2019] R. Gomez-Ojeda, F. Moreno, D. Zuñiga-Noël, D. Scaramuzza, and J. Gonzalez-Jimenez, “Pl-slam: A stereo slam system through the combination of points and line segments,” _IEEE Trans. Robot._ , vol. 35, pp. 734–746, 2019.
  * Yang et al. [2019] Y. Yang, P. Geneva, K. Eckenhoff, and G. Huang, “Visual-inertial odometry with point and line features,” in _Proc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys._ , 2019, pp. 2447–2454.
  * Zhang and Koch [2013] L. Zhang and R. Koch, “An efficient and robust line segment matching approach based on lbd descriptor and pairwise geometric consistency,” _J. of Visual Comm. and Img. Rep._ , vol. 24, pp. 794–805, 2013.
  * Toft et al. [2020] C. Toft, W. Maddern, A. Torii, L. Hammarstrand, E. Stenborg, D. Safari, M. Okutomi, M. Pollefeys, J. Sivic, T. Pajdla, F. Kahl, and T. Sattler, “Long-term visual localization revisited,” _IEEE Trans. Pat. Anal. and Mach. Intell._ , pp. 1–1, 2020.
  * Sarlin et al. [2019] P. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, “From coarse to fine: Robust hierarchical localization at large scale,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2019, pp. 12 716–12 725.
  * Vakhitov and Lempitsky [2019] A. Vakhitov and V. Lempitsky, “Learnable line segment descriptor for visual SLAM,” _IEEE Access_ , pp. 1–1, 2019.
  * Lange et al. [2019] M. Lange, F. Schweinfurth, and A. Schilling, “Dld: A deep learning based line descriptor for line feature matching,” in _Proc. IEEE/RSJ Intl. Conf. on Intell. Robots and Sys._ , 2019, pp. 5910–5915.
  * Lange et al. [2020] M. Lange, C. Raisch, and A. Schilling, “WLD: A Wavelet and Learning based Line Descriptor for Line Feature Matching,” in _Proc. Intl. Symp. on Vis., Mod., and Vis._ , 2020, pp. 39–46.
  * Pautrat et al. [2021] R. Pautrat, L. Juan-Ting, V. Larsson, M. R. Oswald, and M. Pollefeys, “Sold²: Self-supervised occlusion-aware line description and detection,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2021.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Sys. Conf._ , 2017, pp. 5998–6008.
  * Wang et al. [2009] Z. Wang, F. Wu, and Z. Hu, “MSLD: A robust descriptor for line matching,” _Pattern Recognition_ , pp. 941–953, 2009.
  * Lowe [2004] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _Intl. J. of Comput. Vision_ , pp. 91–110, 2004.
  * Tian et al. [2017] Y. Tian, B. Fan, and F. Wu, “L2-net: Deep learning of discriminative patch descriptor in euclidean space,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2017, pp. 6128–6136.
  * Mishchuk et al. [2017] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas, “Working hard to know your neighbor’s margins: Local descriptor learning loss,” in _Advances in Neural Information Processing Sys. Conf._ , 2017, pp. 4826–4837.
  * Yi et al. [2016] K. Yi, E. Trulls, V. Lepetit, and P. Fua, “Lift: Learned invariant feature transform,” in _Proc. European Conf. on Comput. Vision_ , 2016, pp. 467–483.
  * DeTone et al. [2018] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog. Workshops_ , 2018, pp. 224–236.
  * Devlin et al. [2019] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in _Proc. Conf. of the North American Chapter of the Assoc. for Comput. Linguistics: Human Lang. Tech._ , 2019, pp. 4171–4186.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _Proc. Intl. Conf. on Learning Representations_ , 2021\.
  * Wu et al. [2020] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, M. Tomizuka, K. Keutzer, and P. Vajda, “Visual transformers: Token-based image representation and processing for computer vision,” _CoRR_ , vol. abs/2006.03677, 2020.
  * von Gioi et al. [2010] R. G. von Gioi, J. Jakubowicz, J. Morel, and G. Randall, “LSD: A fast line segment detector with a false detection control,” _IEEE Trans. Pat. Anal. and Mach. Intell._ , vol. 32, pp. 722–32, 2010.
  * Wang et al. [2009] L. Wang, U. Neumann, and S. You, “Wide-baseline image matching using line signatures,” in _Proc. IEEE Intl. Conf. on Comput. Vision_ , 2009, pp. 1311–1318.
  * Sarlin et al. [2020] P. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , pp. 4937–4946, 2020.
  * Veličković et al. [2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in _Proc. Intl. Conf. on Learning Representations_ , 2018.
  * Schroff et al. [2015] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2015, pp. 815–823.
  * Radenovic et al. [2018] F. Radenovic, A. Iscen, G. Tolias, Y. Avrithis, and O. Chum, “Revisiting oxford and paris: Large-scale image retrieval benchmarking,” _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , pp. 5706–5715, 2018.
  * Revaud et al. [2019a] J. Revaud, P. Weinzaepfel, C. R. de Souza, N. Pion, G. Csurka, Y. Cabon, and M. Humenberger, “R2D2: repeatable and reliable detector and descriptor,” in _Advances in Neural Information Processing Sys. Conf._ , 2019, pp. 12 405–12 415.
  * Dubrofsky and Woodham [2008] E. Dubrofsky and R. Woodham, “Combining line and point correspondences for homography estimation,” in _Proc. Intl. Symp. Visual Computing_ , vol. 5359, 2008, pp. 202–213.
  * Dai et al. [2017] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2017, pp. 2432–2443.
  * Barnes et al. [2019] D. Barnes, M. Gadd, P. Murcutt, P. Newman, and I. Posner, “The oxford radar robotcar dataset: A radar extension to the oxford robotcar dataset,” _arXiv: 1909.01300_ , 2019.
  * Dusmanu et al. [2019] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-net: A trainable CNN for joint detection and description of local features,” in _Proc. IEEE Conf. on Comput. Vision and Pattern Recog._ , 2019, pp. 8092–8101.
  * Maddern et al. [2017] W. Maddern, G. Pascoe, C. Linegar, and P. Newman, “1 Year, 1000km: The Oxford RobotCar Dataset,” _Intl. J. of Robot. Research_ , vol. 36, no. 1, pp. 3–15, 2017.
  * Revaud et al. [2019b] J. Revaud, J. Almazán, R. S. Rezende, and C. R. de Souza, “Learning with average precision: Training image retrieval with a listwise loss,” in _Proc. IEEE Intl. Conf. on Comput. Vision_ , 2019, pp. 5106–5115.
  * Park et al. [2020] J. Park, K. Joo, Z. Hu, C. Liu, and I. S. Kweon, “Non-local spatial propagation network for depth completion,” in _Proc. European Conf. on Comput. Vision_ , 2020, pp. 120–136.
  * Agostinho et al. [2019] S. Agostinho, J. Gomes, and A. D. Bue, “Cvxpnpl: A unified convex solution to the absolute pose estimation problem from point and line correspondences,” _CoRR_ , 2019.
  * Li et al. [2016] K. Li, J. Yao, M. Lu, Y. Heng, T. Wu, and Y. Li, “Line segment matching: A benchmark,” in _Proc. IEEE Winter Conf. on Appl. of Computer Vision_ , 2016, pp. 1–9.

  *[CV]:
  *[CNN]:
  *[NLP]:
  *[PL-Loc]:
  *[SLAM]:
  *[SFM]:
  *[LBD]:
  *[MSLD]:
  *[LLD]:
  *[WLD]:
  *[BERT]:
  *[MLP]:
  *[MSA]:
  *[LN]:
  *[GNN]:
  *[LSD]:
  *[NN]:
  *[RANSAC]:
  *[GPS]:
  *[ICP]:
  *[PnPL]:
  *[PnP]:
  *[PnL]: