# DeepAAT: Deep Automated Aerial Triangulation for Fast UAV-based Mapping

Zequan Chen, Jianping Li, Member, IEEE, Qusheng Li, Bisheng Yang, Zhen Dong,
Member, IEEE This study was jointly supported by the National Natural Science
Foundation Project (No. 42201477, No. 42130105). (Corresponding author:
Jianping Li) Zequan Chen is with LIESMARS, Wuhan University, Wuhan, China
430079, and Guangzhou Urban Planning & Design Survey Research Institute,
Guangzhou, China 510030. (e-mail<EMAIL_ADDRESS>Jianping Li, Bisheng
Yang and Zhen Dong are with LIESMARS, Wuhan University, Wuhan, China 430079.
(e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Qusheng Li is with Guangxi Zhuang Autonomous Region Remote Sensing Institute
of Nature Resources, Nanning, China 530201 (e-mail<EMAIL_ADDRESS>
###### Abstract

Automated Aerial Triangulation (AAT), aiming to restore image pose and
reconstruct sparse points simultaneously, plays a pivotal role in earth
observation. With its rich research heritage spanning several decades in
photogrammetry, AAT has evolved into a fundamental process widely applied in
large-scale Unmanned Aerial Vehicle (UAV) based mapping. Despite its
advancements, classic AAT methods still face challenges like low efficiency
and limited robustness. This paper introduces DeepAAT, a deep learning network
designed specifically for AAT of UAV imagery. DeepAAT considers both spatial
and spectral characteristics of imagery, enhancing its capability to resolve
erroneous matching pairs and accurately predict image poses. DeepAAT marks a
significant leap in AAT’s efficiency, ensuring thorough scene coverage and
precision. Its processing speed outpaces incremental AAT methods by hundreds
of times and global AAT methods by tens of times while maintaining a
comparable level of reconstruction accuracy. Additionally, DeepAAT’s scene
clustering and merging strategy facilitate rapid localization and pose
determination for large-scale UAV images, even under constrained computing
resources. The experimental results demonstrate DeepAAT’s substantial
improvements over conventional AAT methods, highlighting its potential in the
efficiency and accuracy of UAV-based 3D reconstruction tasks. To benefit the
photogrammetry society, the code of DeepAAT will be released at:
https://github.com/WHU-USI3DV/DeepAAT.

###### Index Terms:

Automated Aerial Triangulation (AAT), Unmanned Aerial Vehicle (UAV), Structure
from Motion (SfM), Orientation

##  I Introduction

Automated Aerial Triangulation (AAT) is a basic task in photogrammetry and
holds substantial research significance [1]. It serves as the initial step in
the 3D reconstruction pipeline of aerial images [2]. AAT’s primary role
involves simultaneously recovering the camera poses and reconstructing sparse
3D points in the scene. These foundational outputs facilitate subsequent dense
image matching and 3D modeling procedures [3]. The derived camera poses and
scene models find diverse applications in digital mapping [4], virtual reality
[5], and smart city [6]. With a research history spanning decades [7],
classical AAT algorithms can be primarily categorized into two groups:
incremental style and global style [8]. Furthermore, the evolution of deep
learning has given rise to numerous supervised AAT algorithms [9]. The
existing AAT methods are reviewed as follows.

###  I-A Classic Automated Aerial Triangulation

The first step of the classic AAT is to perform feature extraction and
matching of all input images. The following steps are different for the global
style and incremental style. The global AAT can predict all camera poses and
scene structure at once [10]. In AAT algorithms, Bundle Adjustment (BA) [11]
is the most time-consuming part. Global AAT only requires to execute BA once,
resulting in higher efficiency. However, it can be difficult to eliminate
outliers, resulting in poor robustness and scene integrity. Incremental AAT
was first proposed by Snavely et al. [12], whose key lies in selecting a good
initial matching image pair [13]. Afterward, incremental AAT adds a new image
to the system sequentially, followed by Perspective-n-Points (PnP) [14],
Triangulation [15], and local BA. Incremental AAT requires multiple BA,
resulting in low reconstruction efficiency in situations with a large number
of images [16]. In addition, due to the accumulation of errors, the
reconstructed scene is prone to drift issues.

Compared to the general scenes, UAV images exhibit distinctive
characteristics, including large volumes, high resolutions, and significant
overlap. Within the realm of classic AAT algorithms, incremental methods have
emerged as the standard approach for UAV image AAT due to their superior
robustness against outliers and ability to provide comprehensive results.
Addressing the challenges posed by large-scale UAV images, most SOTA AAT
methods typically involve employing a divide-and-conquer strategy. This
strategy begins by segmenting the UAV image into blocks based on GPS
information, followed by the fusion of all modules to yield globally
consistent large-scale results. Noteworthy contributions in this field include
the work by Chen et al. [17], which employed the maximum spanning tree to
expand images after dividing the scene map into smaller segments with a
certain degree of overlap, thereby enhancing connectivity and scene map
integrity. Similarly, Xu et al. [18] introduced a hierarchical approach that
constructed a binary tree using images as leaf nodes, subsequently fusing
spatial triads and scenes from the bottom up. This method offers advantages in
terms of robustness, accuracy, and efficiency. Likewise, Bhowmick et al. [19]
initially organized images into hierarchical trees using clustering methods,
then addressed the AAT problem for large-scale images by reconstructing each
small image set and merging them into a common reference framework. Snavely et
al. [20] partitioned extensive scenes by calculating the small bone skeleton
set and reconstructing the skeleton set of the image. This approach reduces
the number of parameters under consideration and enhances reconstruction
efficiency. To sum up, global AAT offers high efficiency but suffers from poor
robustness and scene integrity. On the other hand, incremental AAT exhibits
high robustness and accuracy but tends to have relatively lower time
efficiency.

###  I-B Supervised Automated Aerial Triangulation

Recognizing the limitations encountered by classical AAT algorithms, an
increasing number of studies are exploring the application of deep learning
methods to address these challenges. Many existing deep learning methods
directly regress the depth map and pose of a monocular camera [21], which
usually highly rely on prior information for prediction. In addition, due to
not considering the correlation between depth and pose, the generalization
ability of these networks is limited, making it difficult to obtain ideal
prediction results. BA-Net [22] attempts to use feature metric BA to solve the
AAT problem. It makes end-to-end training possible by designing a
differentiable LM (Levenberg–Marquardt) optimization algorithm, but the LM
algorithm occupies a large amount of memory and has low computational
efficiency. DeepSfM [23] can simultaneously regress the pose and depth maps
corresponding to the image, but it requires coarse poses and depth maps as
initialization, and has high requirements for GPU, making it difficult to
expand to high-resolution images and large-scale environments. DeepMLE [9]
does not require initial values as input, which expresses the two-view AAT
problem as maximum likelihood estimation, learning the relative pose of the
two views by maximizing the likelihood function of correlation. Similarly, for
the problem of binocular vision estimation, Wang et al. [24] proposed a dense
optical flow estimation network for predicting between two frames, which
includes a scale-invariant depth estimation module that can simultaneously
calculate the relative pose of the camera based on the corresponding
relationship of 2D optical flow. DRO [25] is an optimization method based on
recurrent neural networks that iteratively updates camera pose and image depth
to minimize feature measurement errors. Zhuang and Chandraker [26] used a
self-attention graph neural network to enhance strong interactions between
different corresponding relationships and potentially model complex
relationships between points to drive learning. MOAC [27] introduces a grouped
dual cost enhancement module, which enhances the spatial semantic information
and channel relationships of costs, making the optimization more robust to
noise. Moran et al. [28] proposed a new approach to solve AAT problems using
deep learning. They use matched feature points as input, and after permutation
equivariant networks, predict the pose of each camera and 3D points in the
scene. Compared to many existing deep learning AAT methods, it can be applied
to large-scale reconstruction tasks in an unsupervised manner. However, there
are mainly two drawbacks to it. The first one is that it cannot eliminate
incorrectly matched point pairs, which means all the input matched pairs
should be correct and is usually difficult to achieve. Another one is that its
prediction results are still not satisfactory because of the limited
generalizability.

In summary, most of the existing supervised methods can only handle a small
number of low-resolution images, and their regression performance is also
poor, lacking usability and practicality. Hence, the proposed DeepAAT
addresses the existing challenges encountered by both classic and learning-
based AAT algorithms, and presents a meticulously designed deep network
tailored for UAV imagery, emphasizing efficiency, scene completeness, and
practical applicability. The main contributions of this study are threefold:

(1) DeepAAT incorporates a spatial-spectral feature aggregation module,
specifically assimilating both the spatial layout and spectral characteristics
of an image set. This module boosts the network’s ability to perceive the
spatial arrangement of cameras and enhances the global regression capability
for poses.

(2) DeepAAT introduces an outlier rejection module rooted in global
consistency, which effectively generates a reliability evaluation score for
each feature correspondence. This approach facilitates the efficient and
precise elimination of erroneous matching pairs, thereby ensuring accuracy and
reliability throughout the entire 3D reconstruction process.

(3) DeepAAT can efficiently process hundreds of UAV images simultaneously,
marking a significant breakthrough in enhancing the applicability of deep
learning-based AAT algorithms. Furthermore, through a block fusion strategy,
DeepAAT can be effectively scaled up for large-scale scenarios.

The rest of this paper is structured as follows. The preliminaries for our
system are provided in Section II. A brief system overview including the
hardware and software structure is provided in Section III. A detailed
description of DeepAAT is presented in Section IV and experiments are
conducted on UAV image datasets in Section V. Conclusion and future work are
drawn in Section VI.

##  II Preliminary

###  II-A Problem Definition of Automated Aerial Triangulation

The task of AAT refers to estimating the camera poses and 3D feature points
corresponding to the 2D observations on the images. In classic photogrammetry,
it is well studied and understood that the relative camera poses and 3D
feature points can be solved only with the 2D observations [29]. The absolute
camera poses and 3D feature points respective to the geodesic framework can be
then obtained with Ground Control Points (GCPs) or the GPS mounted on the UAV
[30].

Assume that the stationary targeting survey area is viewed by $M$ images,
which are captured by the camera with known pre-calibrated intrinsic parameter
$\mathbf{K}$ at different places along the UAV survey mission. The $M$ unknown
camera poses are represented by a set of projection matrices
$\mathcal{P}=\\{\mathbf{P}_{m}|m=1,...,M\\}$. Each projection matrix
$\mathbf{P}_{m}$ with the size of $3\times 4$ is constructed by camera
rotation $\mathbf{R}_{m}\in SO(3)$ (corresponding quaternion is
$\mathbf{q}_{m}$) and position $\mathbf{t}_{m}\in\mathbb{R}^{3}$ according to
$\mathbf{P}_{m}=[\mathbf{R}_{m}|\mathbf{t}_{m}]$. Given $N$ 3D feature points
in the targeting survey area $\mathcal{F}=\\{\mathbf{F}_{n}|n=1,...,N\\}$,
each 3D feature point is written by
$\mathbf{F}_{n}=[\mathbf{F}^{1}_{n},\mathbf{F}^{2}_{n},\mathbf{F}^{3}_{n},1]^{\top}$
in homogeneous coordinates. If $\mathbf{F}_{n}$ can be observed by the
$m^{th}$ image, its projection on the $m$’s image is given by Eq.(1). As the
depth information $\lambda_{m,n}$ is lost during the projection,
$\mathbf{f}_{m,n}$ is an up-to-scale bearing vector.

$\mathbf{f}_{m,n}=[\mathbf{f}^{1}_{m,n},\mathbf{f}^{2}_{m,n},1]^{\top}=\frac{1}{\lambda_{m,n}}\mathbf{K}\mathbf{P}_{m}\mathbf{F}_{n}.$
(1)

In a typical AAT procedure, the initial step involves 2D feature detection and
matching between pairwise images, which is carried out using the widely used
Scale-Invariant Feature Transform (SIFT) [31] or other robust feature
detectors and descriptors [32]. This step is not the main focus of this work.
Subsequently, a set of 2D feature tracks denoted as
$\mathcal{T}=\\{\mathbf{T}_{n}|n=1,...,N\\}$, is used as the input for our
algorithm. It should be noted that track $\mathbf{T}_{n}$ corresponds to the
3D feature $\mathbf{F}_{n}$ and is constructed by a set of 2D observations
from different images using Eq.(2):

$\mathbf{T}_{n}=\\{\mathbf{f}_{m,n}|m\in\mathcal{J}_{n}\\},$ (2)

where $\mathcal{J}_{n}$ denotes the set of images that can observe the 3D
feature $\mathbf{F}_{n}$. Then the tracks could be used to recover the camera
poses and 3D feature points like the existing incremental [8] or global [33]
AAT strategies.

###  II-B Projective Factorization

Despite the mainstream AAT methods listed above, projective factorization [34]
is also a long-established method in AAT. We provide a brief introduction to
projective factorization, as it forms the foundation for the operation of the
proposed network. The complete image projections, namely the 2D feature tracks
$\mathcal{T}$, can be gathered into a projective matrix
$\mathbf{W}_{\mathbf{prj}}$ in Eq.(3):

$\begin{split}\mathbf{W}_{\mathbf{prj}}&\equiv\left[\begin{matrix}\lambda_{1,1}\mathbf{f}_{1,1}&\lambda_{1,2}\mathbf{f}_{1,2}&\cdots&\lambda_{1,N}\mathbf{f}_{1,N}\\\
\lambda_{2,1}\mathbf{f}_{2,1}&\lambda_{2,2}\mathbf{f}_{2,2}&\cdots&\lambda_{2,N}\mathbf{f}_{2,N}\\\
\vdots&\vdots&\ddots&\vdots\\\
\lambda_{M,1}\mathbf{f}_{M,1}&\lambda_{M,2}\mathbf{f}_{M,2}&\cdots&\lambda_{M,N}\mathbf{f}_{M,N}\\\
\end{matrix}\right]\\\ &=\left[\begin{array}[]{c}\mathbf{KP}_{1}\\\
\mathbf{KP}_{2}\\\ \vdots\\\ \mathbf{KP}_{M}\\\
\end{array}\right]\left[\begin{array}[]{c}\mathbf{F}_{1}\\\ \mathbf{F}_{2}\\\
\vdots\\\ \mathbf{F}_{N}\\\ \end{array}\right]^{\top}.\end{split}$ (3)

If the 3D feature points in the targeting survey area $\mathcal{F}$ are
observed by all the images, the camera poses and 3D feature points can be
recovered using Singular Value Decomposition (SVD) of
$\mathbf{W}_{\mathbf{prj}}$ [34]. For the common cases of missing
observations, the SVD can be replaced with iterative methods [35, 36].
However, these methods are usually too weak for AAT in the presence of
outliers and noise [37], and can not be directly applied to large-scale AAT.
Nevertheless, the formulation of $\mathbf{W}_{\mathbf{prj}}$ provides an ideal
way of keeping spatial correlation information for neural networks.

###  II-C Permutation Equivariant Layer

Let $\mathbf{W}$ be a tensor with the shape of $M\times N\times D$, whose row
represents the image index, the column represents the feature track index and
the third dimension represents the feature index. Taking projective matrix
$\mathbf{W}_{\mathbf{prj}}$ in Eq.(3) as an example,
$\mathbf{W}_{\mathbf{prj}}$ can be rearranged with the shape of $M\times
N\times 2$ (the third dimension records the feature coordinates on the image
plane) to serve as input for the neural network for the sake of convenience.
To recover the camera poses and 3D feature points using a deep neural network,
we expect a particular layer can output the same results irrespective of the
order of the camera poses or the feature tracks. This reordering problem could
be solved using the Permutation Equivariant Layer (PEL)[38], which was first
introduced by Moran et al. [28] to handle SfM problem exploring tensor’s
exchangeability.

Definition 1. Exchangeability of a tensor $\mathbf{W}$ gives rise to the
following property: If we permute arbitrary rows and columns of $\mathbf{W}$,
then feed the permuted $\mathbf{W}$ into a PEL, the output tensor
$\mathbf{W}^{\prime}$ should experience the same permutation of the rows and
columns as illustrated in Fig.1.

Figure 1:  Exchangeability of tensor $\mathbf{W}$.

Theorem 1. [38] Take tensor $\mathbf{W}$ as input, the PEL with five unique
parameters $h_{1}^{(d,o)},h_{2}^{(d,o)},...,h_{4}^{(d,o)}$ and $h_{5}^{(o)}$
could guarantee the output tensor $\mathbf{W}^{\prime}$ with size of $M\times
N\times O$ exchangeable following a specific fully connected layer calculation
rule in Eq. (4), where $d$ and $o$ are the indexes for the input and output
feature channels, respectively.

$\begin{split}W_{m,n}^{\prime(o)}=\sigma\left(\sum_{d=1}^{D}{\left(h_{1}^{\left(d,o\right)}W_{m,n}^{(d)}+\frac{h_{2}^{\left(d,o\right)}}{M}\left(\sum_{m^{\prime}}{W_{m^{\prime},n}^{(d)}}\right)+\frac{h_{3}^{\left(d,o\right)}}{N}\left(\sum_{n^{\prime}}{W_{m,n^{\prime}}^{(d)}}\right)\right.}+\frac{h_{4}^{\left(d,o\right)}}{MN}\left(\sum_{m^{\prime},n^{\prime}}{W_{m^{\prime},n^{\prime}}^{(d)}}\right)+h_{5}^{\left(o\right)}\right)\end{split}$
(4)

Inspired by the initial work proposed by Moran et al. [28], we also utilize
PEL to extract exchangeable high-level geometry correlations from the feature
track matrix $\mathbf{W}$. But different from Moran et al. [28], our proposed
method takes into account not only the geometric features but also the
spectral features of the feature tracks. Furthermore, the outliers in the
feature track matrix are also automatically rejected to enhance the robustness
of the results.

##  III System Overview

The proposed efficient UAV-based mapping system illustrated in Fig.2 is
briefly introduced in this section. As most UAV controllers, such as PixHawk
[39], depend on GPS for trajectory planning and tracking in survey
applications, it is assumed that each UAV image is geotagged with GPS
information provided by the UAV controller. Despite the Single Point
Positioning (SPP) error of GPS potentially reaching 10 meters on the UAV, it
can still serve as a useful guide for the image matching process, focusing on
matching nearby images only, as demonstrated by [8]. To be compatible with
distributed parallel processing and limit the GPU memory usage on one
computing unit for large-scale UAV-based mapping, the proposed system exploits
the hierarchical SfM scheme [17, 18], contains three components, namely, (1)
image clustering, (2) DeepAAT, and (3) cluster merging.

Figure 2:  System overview of the efficient UAV-based mapping system.

(1) Image clustering divides the entire image set into multiple subsets
considering the 2D feature correspondences between images. By treating the
complete image set as a scene graph $\mathbf{G(V,E)}$ [40], each image
represents a vertex in $\mathbf{V}$, and an edge between two image vertices
exists in $\mathbf{E}$ if the two images share feature correspondences.
Setting the number of correspondences between images as the edge weight,
$\mathbf{G(V,E)}$ is segmented using normalized cut [41] iteratively until the
number of images in each subset is within a desired number $N_{subset}$.
$N_{subset}$ can be set according to the GPU memory on each computing unit.

Remark 1. The quantity of 2D feature correspondences between pairs of images
typically serves as a crucial metric for evaluating the reliability of
relative matching. In essence, the greater the number of 2D feature
correspondences between image pairs, the higher their matching reliability,
and conversely, the fewer the matches, the lower the reliability. Our goal is
to achieve a strong level of mutual matching within each subset. Consequently,
the objective of the normalized cut operation on $\mathbf{G(V,E)}$ is to
minimize the sum of edge weights within the cut, while also ensuring a
balanced distribution of elements in each subset to enhance the efficiency of
computing for the following DeepAAT.

(2) DeepAAT efficiently and robustly recovers camera poses and structural
information within each cluster. The network structure and implementation
details of DeepAAT will be described in Section IV.

(3) Cluster merging conducts a global bundle adjustment of the entire images
taking the camera poses from each subset as initial values, hence recovering
the whole camera poses and structures in the targeting survey area. More
specifically, after rejecting the outlier tracks identified by the DeepAAT,
the remaining feature tracks are then re-triangulated using the initial camera
poses resulting from DeepAAT. Then the global bundle adjustment is performed
once to get the final result. Readers can refer to [11] for additional details
on re-triangulation and global bundle adjustment.

##  IV Network Architecture of DeepAAT

The network architecture of DeepAAT mainly consists of three parts, spatial-
spectral feature aggregation module (Section IV-A), global consistency-based
outlier rejecting module (Section IV-B), and pose decode module (Section
IV-C), which are illustrated in Fig.3. The input of DeepAAT includes feature
projection matrix $\mathbf{W}_{prj}$ constructed by reordering Eq.(3) with the
shape of $M\times N\times 2$, SIFT feature descriptor matrix
$\mathbf{W}_{des}$ with the shape of $M\times N\times 128$, and the geotag
matrix $\mathbf{W}_{gps}$ with the shape of $M\times 3$. It’s important to
highlight that the GPS-derived latitude, longitude, and altitude values are
initially transformed into the East-North-Up (ENU) local coordinate system
[42]. Following this transformation, they are further normalized to enhance
network generativity. By feeding the input into the spatial-spectral feature
aggregation module, a high-level embedded feature $\mathbf{W}_{em}$ is
extracted for the downstream tasks. Then, the global consistency-based outlier
rejecting module detects the outliers in the projection matrix
$\mathbf{W}_{prj}$ and gives the confidence for each 2D observation using
$\mathbf{W}_{outlier}$. Meanwhile, the pose decode module recovers the camera
poses. The loss functions will be detailed in Section IV-D.

Figure 3:  Network architecture of DeepAAT.

###  IV-A Spatial-Spectral Feature Aggregation Module

Position Encoding: The coordinates within the feature projection matrix
$\mathbf{W}_{prj}$ and the GPS matrix $\mathbf{W}_{gps}$ are both essential
for the network to comprehend the spatial distribution. Position encoding [43]
has been employed for both of these location-related pieces of information to
enhance the network’s ability to distinguish relative positional relationships
among input data, which is written as follows:

$\footnotesize\begin{split}\varepsilon(x)=[sin(2^{0}\pi x),cos(2^{0}\pi
x),...,sin(2^{L-1}\pi x),cos(2^{L-1}\pi x)]^{\top},\end{split}$ (5)

where $x$ is the coordinate value in each dimension, $L$ is the coding level.
Position encoding solely influences the last dimension of $\mathbf{W}_{prj}$
and $\mathbf{W}_{gps}$. As for the GPS matrix $\mathbf{W}_{gps}$, to ensure
its dimensional consistency with $\mathbf{W}_{prj}$ and $\mathbf{W}_{des}$,
the proposed method conducts a dimension expansion operation in the first
dimension (image indexes), transforming it from a two-dimensional matrix into
a three-dimensional one. Position encoding does not have learnable parameters,
but it enhances the network’s ability to distinguish location information.

Residual Permutation Equivariant Layer: The residual PEL employed in this
paper consists of a consecutive pair of PEL, Instance Normalization (IN) [44],
and Rectified Linear Unit (ReLU) combinations. Within the residual PEL, input
and output data are summated through skip connections [45], serving two
purposes: (1) ensuring stable gradient propagation within the network; and (2)
facilitating the fusion of shallow layers, which contain more actual
positional information, with deeper layers that offer more distinctive and
discriminative feature information. It’s worth noting that IN and ReLU do not
alter the permutation equivariant properties of PEL.

###  IV-B Global Consistency-based Outlier Rejecting Module

Even with pair-wise Epipolar geometry verification, $\mathbf{W}_{prj}$ still
contain a substantial number of outlier matches, which can significantly
impact obtaining correct global triangulation results and ensuring the proper
convergence of the BA. Therefore, the proposed method utilizes a global
consistency outlier rejecting module, which, by leveraging global information
from the embedded feature $\mathbf{W}_{em}$, ensures that the subsequent BA
operates successfully.

The global consistency-based outlier rejecting module primarily comprises
three consecutive projection layers. Following the first two projection layers
are the Context Normalization (CN) and ReLU activation functions. The CN
primarily serves to integrate data, allowing the previous layer’s output to
acquire corresponding context information in both camera poses and feature
tracks. This aids the network in identifying outliers. Following the last
projection layer is the Sigmoid. After passing through the Sigmoid function,
the network’s output is a score matrix with values ranging from 0 to 1, with
dimensions $M\times N$. For each non-empty point, the score represents the
probability that each 2D feature is an inlier. The closer the score is to 1,
the higher the confidence that it is an inlier. In the actual outlier
detection process, with a given threshold $\tau_{outlier}$, the predicted
scores greater than $\tau_{outlier}$ are considered as inliers, while those
with scores less than $\tau_{outlier}$ are considered outliers.

###  IV-C Pose Decode Module

The pose decode module utilizes the global information encoded by the spatial-
spectral feature aggregation module to decode the camera’s position and
orientation. The decoder first performs mean pooling on the input feature
along the column dimension (feature tracks), which maps the embedded feature
$\mathbf{W}_{em}$ with the shape of $M\times N\times O_{em}$ to $M\times
O_{em}$. The reason for choosing mean pooling is that each camera observes a
different number of 3D points in the input scene. By averaging the features
observed by each camera, it can fairly represent the general characteristics
of the scene observed by the $M$ cameras, regardless of the number of 3D
points observed. This enables the decoder to obtain the global context
information for each camera within the scene.

Camera position decoder: In the camera position decoding branch, GPS location
information is reintroduced to improve the decoder’s spatial positioning
awareness. Additionally, the network performs regression on the camera’s
position offsets, which reflect the errors in the GPS tags, other than the
camera’s position directly. This is done because the magnitudes of GPS errors
in each direction consistently fall within a specific range.

Camera rotation decoder: In the camera rotation decoding branch, the
quaternion of each camera is regressed with two perception layers.

###  IV-D Loss Function

The loss function for DeepAAT comprises three components:
$\mathcal{L}_{outlier}$, $\mathcal{L}_{position}$, and
$\mathcal{L}_{rotation}$, which written by:

$\begin{split}\mathcal{L}=\mathcal{L}_{outlier}+\alpha\mathcal{L}_{position}+\beta\mathcal{L}_{rotation},\end{split}$
(6)

where $\alpha$ and $\beta$ are balance factors. $\mathcal{L}_{outlier}$ is a
Binary Cross-Entropy (BCE) like loss used to supervise the global consistency-
based outlier rejecting module:

$\begin{split}\mathcal{L}_{outlier}=-\frac{1}{MN}\sum_{m=0}^{M-1}{\sum_{n=0}^{N-1}{\big{(}\mathbf{W}^{m,n}_{outlier}\mathbf{log}(\hat{\mathbf{W}}^{m,n}_{outlier})+(1-\mathbf{W}^{m,n}_{outlier})\mathbf{log}(1-\hat{\mathbf{W}}^{m,n}_{outlier})}\big{)}},\end{split}$
(7)

where $\hat{\mathbf{W}}^{m,n}_{outlier}$ is the predicted outlier score range
from $[0,1]$. Both the rotation loss $\mathcal{L}_{rotation}$ and translation
loss $\mathcal{L}_{position}$ are implemented using the mean square loss
function:

$\begin{split}\mathcal{L}_{rotation}=\frac{1}{M}\sum_{m=0}^{M-1}{||\mathbf{q}_{m}-\hat{\mathbf{q}}_{m}||_{2}},\\\
\mathcal{L}_{position}=\frac{1}{M}\sum_{m=0}^{M-1}{||\mathbf{t}_{m}-\hat{\mathbf{t}}_{m}||_{2}},\end{split}$
(8)

where $\hat{\mathbf{q}}_{m}$ and $\hat{\mathbf{t}}_{m}$ are the predicted
rotation and translation for the $m^{th}$ camera.

##  V Experiments

###  V-A Dataset

The experimental data, as depicted in Fig.4, was collected from an urban area
including complex road networks, hills, construction sites, and buildings. A
total of 4,992 UAV images were employed, subdivided into eight uniformed
blocks labeled A to H. These images underwent preprocessing through SfM with
Ground Control Points (GCPs) to establish reference data. Throughout the
experiments, data from blocks A to G were used for training, while data from
block H was employed as testing data.

Figure 4:  UAV-based image dataset used for the experiments. The dataset is
divided into eight blocks.

###  V-B Implementation Details

####  V-B1 Training sample generation

The used training samples were generated through random sampling of images
within each block. In the context of UAV AAT, typically, images that are
closer to each other tend to have a greater number of feature correspondences
and exhibit more stable connectivity. Specifically, for a given image set, one
image is randomly selected to serve as the central image, denoted as
$\mathbf{I}_{c}$. Then, according to the GPS position, Euclidean distances
from all the other images to $\mathbf{I}_{c}$ are calculated and arranged in
ascending order. Finally, based on the provided minimum and maximum sampling
image limits, $N_{min}$ and $N_{max}$, a random number of sampled images,
$N_{c}$, is determined. The $N_{c}$ closest images to $\mathbf{I}_{c}$ are
selected to generate the sample data. Using this sampling strategy, a vast
number of distinct samples can be generated. For instance, $N_{min}$ and
$N_{max}$ are set as 100 and 130. A single image set can generate a total of
$624\times(N_{max}-N_{min})=18,720$ samples.

####  V-B2 Data augmentation

To enhance the network’s generalizability, data augmentation is applied to the
training data, focusing on two main aspects. Firstly, Gaussian noise with a
mean of zero and a standard deviation of 0.01 was added to the input
$\mathbf{W}_{prj}$. Secondly, given the limited amount of training data, and
to augment and leverage the network’s permutation equivariance capability,
random row and column permutations were applied to the training samples during
training.

####  V-B3 Parameter settings

The experiments involve the configuration of certain hyperparameters.
Specifically, regarding position encoding, the encoding order is set at $L=4$.
In the spatial-spectral aggregation module, the embedded feature dimension is
set at $O_{em}=256$. In the outlier rejecting module, the outlier detection
threshold is set at $\tau_{outlier}=0.5$. In the configuration of weights
within the loss function, the parameters $\alpha$ and $\beta$, which govern
the weights for rotation and translation, are assigned values of 0.9 and 0.1,
respectively. This allocation stems from the observation that, in contrast to
directly predicted rotation, the prediction of translation is effectively an
adjustment relative to the initial estimate. Consequently, it is appropriate
to assign a lesser weight to translation compared to rotation.

####  V-B4 Evaluation criteria

This paper evaluates the experimental results from various perspectives. For
the outlier removal results, evaluation metrics commonly used in binary
classification tasks are employed, including $Accuracy$, $Precision$,
$Recall$, and $F_{1}$ score, which are calculated as follows:

$\begin{split}Accuracy&=\frac{TP+TN}{TP+FP+TN+FN},\\\
Precision&=\frac{TP}{TP+FP},\\\ Recall&=\frac{TP}{TP+FN},\\\
F_{1}&=\frac{2\times Precision\times Recall}{Precision+Recall},\end{split}$
(9)

where $TP$ is True Positive, $FP$ is False Positive, $TN$ is True Negative,
$FN$ is False Negative.

For the reconstruction results, we use the reprojection error $E_{repro}$,
position error $E_{t}$ and angle error $E_{R}$ to evaluate from three aspects.

$\begin{split}\left\\{\begin{matrix}{E_{repro}=\frac{1}{n_{2d}}{\sum\limits_{i=1}^{m}{\sum\limits_{j=1}^{n}\left\|\left({x_{ij}^{1}-\frac{P_{i}^{1}X_{j}}{P_{i}^{3}X_{j}},x_{ij}^{2}-\frac{P_{i}^{2}X_{j}}{P_{i}^{3}X_{j}}}\right)\right\|_{2}}}},\\\
{E_{t}=\frac{1}{m}{\sum\limits_{i=1}^{m}\left\|{{\hat{t}}_{i}-{\overset{\sim}{t}}_{i}}\right\|_{2}}},\\\
{E_{R}=\frac{1}{m}{\sum\limits_{i=1}^{m}{\cos^{-1}\left({\frac{1}{2}\left({tr\left({{\hat{R}}_{i}^{T}{\overset{\sim}{R}}_{i}}\right)-1}\right)}\right)}}},\end{matrix}\right.\end{split}$
(10)

where, $n_{2d}$ represents the number of 2D pixels in the scene, $m$ is the
number of cameras, $n$ is the number of 3D points, and $x^{k}_{ij}$ represents
the $k$th dimension of the coordinate of the $j$th 3D point observed by the
$i$th image, $P_{i}^{k}$ represents the $k$th row of the $i$th camera matrix,
$X_{j}$ represents the coordinate of the $j$th 3D point, $\hat{t}_{i}$
represents the reference value of camera position, $\widetilde{t}_{i}$
represents the predicted camera position, $\hat{R}_{i}$ represents the
reference value of camera rotation, $\widetilde{R}_{i}$ represents the
predicted camera rotation value, and $tr()$ represents the trace of the matrix
(i.e. the sum of the main diagonal elements of the matrix).

In addition, we also record the time used in network prediction and compare it
with baseline methods as an important evaluation indicator.

####  V-B5 Computation resources

The configuration of the machine used in our experiments is as follows. CPU:
Intel (R) Xeon (R) Silver 4210R CPU @ 2.40GHz, GPU: NVIDIA A100 SXM4 80GB. To
control the memory size and computational complexity, all network training and
prediction tasks involved in this article can be run on a single Tesla V100
with 32GB memory.

###  V-C Results of scene segmenting and merging

To facilitate large-scale reconstruction tasks, we employed a strategy of
image clustering and merging. During the image clustering phase, we set a
maximum limit of 100 cameras for each subset. Consequently, block H was
ultimately divided into 8 distinct blocks, with each subset containing between
72 to 95 cameras. The clustering outcome is illustrated in Fig.5, where each
circle denotes a camera, and different colors represent individual subsets.
Notably, the gray lines in the figure are the severed edges that connect
cameras across different subsets.

Figure 5:  Image clustering result, where the number after ”_” represents the
number of cameras included in the subset. TABLE I: Outlier rejection result

Scene | ↑Acc | ↑Pre | ↑Rec | ↑F1 | Time/s
---|---|---|---|---|---
H1_95 | 0.959 | 0.966 | 0.980 | 0.973 | 0.820
H2_73 | 0.933 | 0.960 | 0.952 | 0.956 | 0.087
H3_79 | 0.947 | 0.966 | 0.968 | 0.967 | 0.094
H4_76 | 0.951 | 0.960 | 0.974 | 0.967 | 0.085
H5_72 | 0.967 | 0.978 | 0.982 | 0.980 | 0.117
H6_82 | 0.965 | 0.980 | 0.979 | 0.980 | 0.129
H7_75 | 0.952 | 0.975 | 0.966 | 0.970 | 0.092
H8_72 | 0.967 | 0.981 | 0.980 | 0.980 | 0.080
Mean | 0.955 | 0.971 | 0.973 | 0.972 | 0.188

  * •

*Acc(Accuracy), Pre(Precision), Rec(Recall), F1(F1 Score)

Table I shows that the average performance of the outlier rejection in the
scene surpasses 0.95 across all four metrics, with a notable recall rate of
0.973. This indicates that about 97.3% of the pixels identified by the network
as positive are indeed true positives. Such high accuracy is advantageous for
the subsequent steps of global triangulation and BA. This also demonstrates
that the global consistency-based outlier rejection module, as designed in
this paper, is highly effective and applicable throughout the entire
algorithmic process. Moreover, for clustered scenes, the network’s prediction
time is under one second, highlighting the proposed network’s high operational
efficiency in AAT tasks. Here, we predict all 8 scenarios through a single
model loading, and except for the first scenario, the remaining 7 scenarios do
not require reloading the model, resulting in a nearly tenfold reduction in
time consumption.

TABLE II: Pose prediction result

Scene | IPE/m | DeepAAT | Results after BA
---|---|---|---
RPE/pix | PE/m | RE/° | RPE/pix | PE/m | RE/°
H1_95 | 5.153 | 64.495 | 4.170 | 1.961 | 0.490 | 2.732 | 0.032
H2_73 | 5.159 | 52.222 | 4.495 | 1.322 | 0.459 | 1.865 | 0.048
H3_79 | 5.192 | 60.994 | 4.941 | 1.845 | 0.475 | 1.994 | 0.032
H4_76 | 4.955 | 52.681 | 3.868 | 1.903 | 0.467 | 2.570 | 0.023
H5_72 | 5.259 | 45.699 | 4.184 | 1.155 | 0.487 | 2.583 | 0.027
H6_82 | 5.417 | 59.702 | 4.376 | 2.042 | 0.464 | 1.938 | 0.027
H7_75 | 5.262 | 53.557 | 4.530 | 1.897 | 0.477 | 1.801 | 0.034
H8_72 | 5.091 | 45.062 | 4.200 | 1.813 | 0.486 | 2.375 | 0.028
Mean | 5.186 | 54.302 | 4.346 | 1.742 | 0.476 | 2.232 | 0.031

  * •

*IPE(Initial Position Error), RPE(Reprojection Error), PE(Position Error), RE(Rotation Error)

Table II demonstrates that across all eight clustered scenes, the predicted
camera position error is consistently lower than the initial position error.
The results show greater precision in predicting rotation, with an average
error of less than 2°, which is highly beneficial for accurate subsequent
adjustments. After BA, the average reprojection error across all scenes is
less than 0.5 pixels, and the rotation error is under 0.1°. The visualized
results, as depicted in Fig.6, further affirm the effectiveness of the
algorithm.

Figure 6:  Network prediction results (upper) and results after BA (lower).

In the following, the Cluster Merging algorithm described in Section III is
used to fuse the above 8 segmented scenes, and the results are shown in Fig.7
and Tab.III. From Fig.7, it can be seen that after using GPS information to
globally align all segmented scenes, there is a significant offset between
different scenes. After Cluster Merging, the inconsistency between scenes was
effectively eliminated, resulting in globally consistent fusion results that
were similar in appearance to the reference one. From Tab.III, it can be seen
that the reprojection error of the scene has slightly decreased compared to
the average reprojection error of the segmented scene, but in terms of
position error and rotation error, it has slightly increased compared to the
average result of the segmented scene. In addition, the reconstructed scene
points have a slight increase compared to the 3D points of the reference
scene. These results indicate that the proposed hierarchical AAT scheme can
effectively complete large-scale AAT tasks.

Figure 7:  (a) GPS alignment result, (b) cluster merging result, and (c) reference result. TABLE III: Cluster merging result RPE/pix | PE/m | RE/° | Points | Reference points
---|---|---|---|---
0.471 | 2.726 | 0.450 | 553164 | 549517

###  V-D Comparison

We compare the proposed algorithm with the SOTA methods, including:

ESFM [28]: A neural network architecture is proposed, which takes track points
as input in the form of a matrix. It can simultaneously predict camera pose
and scene points, and use reprojection error as the loss function.

Colmap [8]: A state-of-the-art open-source incremental SfM pipeline library,
widely used in pose estimation and scene reconstruction tasks. Colmap provides
both UI interface and command line running mode, making it easy to operate and
has good reconstruction results.

OpenMVG [46]: Provides both incremental SfM and global SfM implementations,
with global SfM being the current SOTA in the open-source library. By using
the command line, step-by-step SfM can be easily implemented.

Because the output of ESFM and the proposed method is the result before BA, we
directly compared the predicted results of the two networks, including
reprojection error and rotation error. The results are shown in Tab.IV, and
some prediction scenarios are shown in Fig.8. To make the learning space of
ESFM more consistent, we standardize all input scenes, so that the network
learns the relative positions of all cameras relative to the first camera in
the scene. When comparing the proposed DeepAAT with Colmap and OpenMVG, we
directly compared the final reconstruction results after BA, including
reprojection error, final scene points, and time consumption. The results are
shown in Tab.V, and the predicted scenes are shown in Fig.9.

TABLE IV: Comparison of ESFM and the proposed DeepAAT

Scene | ESFM | DeepAAT (ours)
---|---|---
↓RPE/pixel | ↓PE/m | ↓RE/° | ↓RPE/pixel | ↓PE/m | ↓RE/°
1_128 | 410.580 | 229.447 | 62.265 | 46.891 | 4.340 | 1.610
2_107 | 460.225 | 207.427 | 103.297 | 58.965 | 3.807 | 2.409
3_112 | 356.813 | 225.512 | 55.177 | 46.582 | 3.995 | 1.422
4_104 | 442.035 | 209.172 | 101.788 | 58.436 | 4.301 | 2.363
5_104 | 440.410 | 197.034 | 92.420 | 41.844 | 3.962 | 2.091
6_106 | 457.345 | 204.852 | 59.944 | 49.994 | 4.087 | 1.713
7_118 | 397.622 | 205.955 | 60.104 | 43.848 | 3.957 | 1.631
8_127 | 433.450 | 215.331 | 68.668 | 45.161 | 4.177 | 1.417
9_127 | 362.216 | 220.011 | 95.587 | 49.067 | 4.455 | 1.676
10_129 | 479.165 | 223.690 | 66.945 | 44.770 | 4.144 | 1.483
mean | 423.986 | 213.843 | 76.619 | 48.556 | 4.123 | 1.781

Figure 8:  Comparison of partial scenarios predicted by ESFM (Upper) and
DeepAAT (Lower).

From Tab.IV, it can be seen that the prediction results of the proposed
DeepAAT are much smaller than ESFM in terms of reprojection error, position
error, and rotation error. From the four comparative scenarios in Fig.8, it
can be seen that the scenarios predicted by the proposed DeepAAT are all
correct, but the prediction results of ESFM are relatively chaotic, resulting
in the inability to obtain correct results through subsequent global BA. The
main reasons are as follows:

(1) DeepAAT integrates GPS information as a prior, significantly enhancing the
network’s spatial awareness and location perception. Crucially, DeepAAT
predicts the offset in camera position, which is a relative measure to GPS
coordinates, rather than attempting to directly ascertain the precise location
of each camera.

(2) Unlike ESFM, DeepAAT incorporates a global consistency-based outlier
rejection module, which effectively eliminates erroneous matching points that
persist even after geometric verification. As a result, the prediction
outcomes produced by DeepAAT are considerably more refined and cleaner. In
contrast, ESFM lacks a denoising feature, and the presence of noise points in
its framework can adversely affect the network’s ability to accurately learn
and represent the correct scene.

TABLE V: Comparison results between DeepAAT and traditional algorithms

Scene | Colmap | OpenMVG Incremental | OpenMVG Global | DeepAAT (Ours)
---|---|---|---|---
↓RPE/pix | ↓Time /s | ↓RPE/pix | ↓Time/s | ↓RPE/pixel | ↓Time/s | ↓RPE/pix | ↓Time/s
1_128 | 0.500 | 465.966 | 0.478 | 548.536 | 0.548 | 29.651 | 0.489 | 0.845
2_107 | 0.566 | 296.569 | 0.518 | 390.071 | 0.611 | 15.830 | 0.482 | 0.832
3_112 | 0.501 | 377.211 | 0.479 | 601.795 | 0.539 | 28.098 | 0.472 | 0.861
4_104 | 0.554 | 276.811 | 0.522 | 418.626 | 0.611 | 14.442 | 0.491 | 0.798
5_104 | 0.528 | 285.427 | 0.473 | 367.106 | 0.590 | 13.879 | 0.447 | 0.780
6_106 | 0.506 | 365.869 | 0.450 | 417.672 | 0.516 | 19.331 | 0.484 | 0.815
7_118 | 0.507 | 375.640 | 0.478 | 564.906 | 0.550 | 25.276 | 0.467 | 0.831
8_127 | 0.521 | 437.120 | Reconstruction failure | 0.548 | 34.464 | 0.473 | 0.862
9_127 | 0.478 | 439.329 | 0.481 | 423.064 | 0.551 | 22.412 | 0.483 | 0.859
10_129 | 0.503 | 465.138 | 0.458 | 615.676 | 0.522 | 28.488 | 0.487 | 0.868
Mean | 0.516 | 378.508 | 0.482 | 484.050 | 0.559 | 23.187 | 0.478 | 0.835

Table V reveals that DeepAAT outperforms all other methods in terms of average
reprojection error and average time consumption, with both indicators
surpassing those of Colmap and OpenMVG. Its most striking advantage lies in
time efficiency, where DeepAAT significantly outpaces the comparison methods
across all test scenarios. Specifically, DeepAAT’s average reconstruction
efficiency is 453 times greater than Colmap, 580 times that of OpenMVG
Incremental, and 28 times that of OpenMVG Global. This suggests that the
proposed network substantially enhances the efficiency of AAT reconstruction
while maintaining scene integrity. Concurrently, it also improves
reconstruction accuracy to a certain degree. As indicated in Table V, OpenMVG
Incremental failed to reconstruct scene 8_127, likely due to the stringent
requirements of the incremental SfM algorithm on initial image pair selection
and the relative instability of the OpenMVG Incremental algorithm.

###  V-E Ablation Study

To test the impact of the core modules proposed in DeepAAT, the following
ablation experiments were conducted: ① The encoding order of GPS $L_{G}$ and
the projection matrix $L_{prj}$ is set to 2; ② The encoding order of GPS
$L_{G}$ and the projection matrix $L_{prj}$ is set to 4, which is the setting
used in this article; ③ The encoding order of GPS $L_{G}$ and the projection
matrix $L_{prj}$ is set to 6; ④ Remove the spatial spectral feature
aggregation module, similar to ESFM, using only the projection matrix
$\mathbf{W}_{prj}$ as the network input; ⑤ Remove the global consistency-based
outlier rejecting module. Here, the predicted pose of the network is directly
used to triangulate all matching track points during global triangulation.
These experimental data are the average of 10 test data results, which are
shown in Tab.VI. From Tab.VI, it can be seen that:

TABLE VI: Results of ablation study

Setting | Outlier rejection | Pose estimation
---|---|---
↑Acc | ↑Pre | ↑Rec | ↑F1 | ↓RPE/pix | ↓PE/m | ↓RE/°
① | 0.967 | 0.973 | 0.987 | 0.980 | 49.460 | 3.974 | 2.411
② | 0.966 | 0.974 | 0.985 | 0.979 | 48.556 | 4.123 | 1.781
③ | 0.967 | 0.971 | 0.989 | 0.980 | 57.947 | 4.953 | 2.036
④ | 0.964 | 0.970 | 0.986 | 0.978 | 98.414 | 5.104 | 2.990
⑤ | / | / | / | / | 48.599 | 4.123 | 1.781

(1) Encoding order $L$ of GPS information and the projection matrix in the
spatial-spectral feature aggregation module has little impact on the
experimental results, indicating that as long as the encoding order is set
within a reasonable range, good experimental results can be achieved.

(2) Upon removal of the spatial-spectral feature aggregation module, there was
a marked decline in the network’s overall performance, particularly notable in
pose prediction tasks. The reprojection error more than doubled compared to
the optimal outcome, accompanied by substantial deviations in both position
and rotation errors. These experimental findings highlight that the
integration of GPS positioning data and feature point descriptor into the
network input plays a critical role in significantly enhancing network
performance.

(3) Upon removal of the global consistency-based outlier rejecting module,
there is an increase in the reprojection error. As shown in Fig.9, the noise
feature points generated by triangulation increases significantly. The
outliers included in the scene can have a negative impact on subsequent BA and
easily lead to local optimum. For example, for scene 8_127, the final result
after BA is shown in Fig.10. It can be seen that there is a camera (marked
with a green dashed box) whose pose has been optimized erroneously due to the
influence of outliers.

Figure 9:  Prediction Scenarios without outlier rejecting (Left) and with
outlier rejecting (Right). Figure 10: BA results without outlier filtering
(Left) and with outlier filtering (Right).

###  V-F Generalization experiments on different sizes of input images

The proposed network does not require a fixed number of images as input.
Therefore, to test the generalization of the network in scenarios with
inconsistent camera numbers compared to the training sample, we apply the
model trained on scene from 100-130 images to predict scenes with 30-50
cameras and 400-430 cameras. The predicted results are shown in Tab.VII and
Tab.VIII.

TABLE VII: Results of outlier rejecting for 100-130 camera models in different
numbers of image prediction tasks

Number of images | Scene | ↑Acc | ↑Pre | ↑Rec | ↑F1 | ↓Time/s
---|---|---|---|---|---|---
30-50 | 1_45 | 0.952 | 0.968 | 0.973 | 0.971 | 0.720
2_42 | 0.964 | 0.976 | 0.981 | 0.979 | 0.713
3_50 | 0.950 | 0.961 | 0.976 | 0.969 | 0.749
4_34 | 0.920 | 0.953 | 0.947 | 0.950 | 0.698
5_35 | 0.942 | 0.970 | 0.955 | 0.962 | 0.699
Mean | 0.946 | 0.966 | 0.967 | 0.966 | 0.716
400-430 | 1_420 | 0.971 | 0.973 | 0.99 | 0.982 | 1.186
2_428 | 0.976 | 0.977 | 0.993 | 0.985 | 1.188
3_406 | 0.977 | 0.979 | 0.993 | 0.986 | 1.148
4_401 | 0.975 | 0.976 | 0.993 | 0.985 | 1.152
5_404 | 0.977 | 0.978 | 0.994 | 0.986 | 1.139
Mean | 0.975 | 0.977 | 0.993 | 0.985 | 1.163

TABLE VIII: Pose prediction results of the 100-130 camera trained model in
different numbers of camera prediction tasks

Cam num | Scene | IPE/m | Results of AAT | Results after BA
---|---|---|---|---
↓RPE/pix | ↓PE/m | ↓RE/° | ↓RPE/pix | ↓PE/m | ↓RE/°
30-50 | 1_45 | 5.309 | 56.545 | 4.719 | 2.055 | 0.466 | 2.144 | 0.032
2_42 | 4.951 | 48.010 | 4.071 | 1.221 | 0.473 | 2.291 | 0.029
3_50 | 5.245 | 50.903 | 3.983 | 1.701 | 0.475 | 2.208 | 0.038
4_34 | 4.859 | 52.996 | 4.562 | 2.386 | 0.447 | 1.172 | 0.035
5_35 | 5.448 | 73.172 | 4.393 | 2.586 | 0.415 | 1.697 | 0.030
Mean | 5.162 | 56.325 | 4.346 | 1.990 | 0.455 | 1.902 | 0.033
400-430 | 1_420 | 5.009 | 63.770 | 5.512 | 1.922 | 0.475 | 3.132 | 0.046
2_428 | 5.111 | 56.347 | 4.928 | 1.884 | 0.482 | 3.023 | 0.029
3_406 | 5.046 | 54.624 | 4.774 | 1.878 | 0.483 | 3.164 | 0.028
4_401 | 4.919 | 59.247 | 4.905 | 1.872 | 0.486 | 3.270 | 0.041
5_404 | 4.935 | 59.912 | 5.254 | 1.862 | 0.486 | 3.148 | 0.034
Mean | 5.004 | 58.780 | 5.075 | 1.884 | 0.482 | 3.147 | 0.036

Figure 11: Pose prediction results of the 100-130 camera trained model in
different numbers of camera prediction tasks

The prediction results demonstrate that the proposed DeepAAT is versatile,
excelling not only in scenes with a similar number of images but also in
scenarios with significantly more or fewer images. As indicated in Tab.VII,
the network trained on scenes with 100-130 images shows a decline in accuracy,
precision, recall, and F1 score when applied to 30-50 image scenes.
Conversely, its performance metrics improve in 400-430 image scenes. This
improvement could be attributed to the longer average track length of matching
points in scenes with more cameras, increasing the likelihood of points being
classified as inliers and reducing false negatives. This explains the high
recall of 0.993 in scenes with 400-430 cameras. Regarding prediction time, the
network demonstrates a notable efficiency: despite a rapid increase in the
number of cameras, the time required for network prediction increases only
marginally. This efficiency is a significant advantage in practical
applications. In traditional AAT algorithms, both incremental and global, time
consumption escalates quickly with an increasing number of scene images, a
trend particularly pronounced in incremental SfM. Thus, our network’s time-
saving benefits become more pronounced with larger sets of scene images.

From Tab.VIII, we observe that the network, when trained on scenes with
100-130 images, exhibits a slight increase in average reprojection error,
position error, and rotation error while predicting scenes with 30-50 images
and those with 400-430 images. However, in scenarios such as 1_420 and 5_404,
the network’s predicted position errors surpass the initial position errors.
Nevertheless, following global BA, all scenes achieve accurate reconstruction
results, as illustrated in Fig.11. This outcome not only underscores the
network’s effective scene initialization capabilities but also highlights its
robust reconstruction prowess.

This experimental outcome indicates the superior performance of the proposed
DeepAAT. It exhibits the capability to effectively handle scene prediction
tasks several times larger than its training scope on smaller scenes.
Typically, GPU memory consumption is higher during the training phase than in
testing for most deep-learning tasks. Consequently, this characteristic
significantly enhances the practicality of DeepAAT, making it a robust
solution for large-scale applications.

##  VI Conclusion

AAT of UAV images has gained widespread adoption in 3D reconstruction, favored
for its flexibility and cost-effectiveness. However, challenges persist:
incremental AAT methods struggle with low reconstruction efficiency, global
AAT methods grapple with subpar robustness and scene integrity, and deep
learning-based algorithms often falter when processing a vast number of
images. To overcome these challenges, we introduce DeepAAT, a novel approach
designed to enhance the efficiency of UAV AAT while maintaining the accuracy
and completeness of the reconstructed scenes. Our experiments demonstrate that
DeepAAT’s time efficiency outstrips incremental algorithms by hundreds of
times and global algorithms by tens of times. In the near future, we will
extend DeepAAT to the image set without GPS information.

## References

  * Tanathong and Lee [2014] S. Tanathong and I. Lee, “Using gps/ins data to enhance image matching for real-time aerial triangulation,” _Computers & Geosciences_, vol. 72, pp. 244–254, 2014.
  * Zhong et al. [2023] J. Zhong, J. Yan, M. Li, and J.-P. Barriot, “A deep learning-based local feature extraction method for improved image matching and surface reconstruction from yutu-2 pcam images on the moon,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 206, pp. 16–29, 2023.
  * Jiang et al. [2021] S. Jiang, W. Jiang, and L. Wang, “Unmanned aerial vehicle-based photogrammetric 3d mapping: A survey of techniques, applications, and challenges,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 10, no. 2, pp. 135–171, 2021.
  * Hasheminasab et al. [2022] S. M. Hasheminasab, T. Zhou, Y.-C. Lin, and A. Habib, “Linear feature-based triangulation for large-scale orthophoto generation over mechanized agricultural fields,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 60, pp. 1–18, 2022.
  * Jiang et al. [2020] S. Jiang, C. Jiang, and W. Jiang, “Efficient structure from motion for large-scale uav images: A review and a comparison of sfm tools,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 167, pp. 230–251, 2020.
  * Zhou et al. [2020] G. Zhou, X. Bao, S. Ye, H. Wang, and H. Yan, “Selection of optimal building facade texture images from uav-based multiple oblique image flows,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 2, pp. 1534–1552, 2020.
  * Schenk [1997] T. Schenk, “Towards automatic aerial triangulation,” _ISPRS Journal of Photogrammetry and remote Sensing_ , vol. 52, no. 3, pp. 110–121, 1997.
  * Schonberger and Frahm [2016] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4104–4113.
  * Xiao et al. [2022] Y. Xiao, L. Li, X. Li, and J. Yao, “Deepmle: A robust deep maximum likelihood estimator for two-view structure from motion,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 10 643–10 650.
  * Govindu [2004] V. M. Govindu, “Lie-algebraic averaging for globally consistent motion estimation,” in _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , vol. 1. IEEE, 2004, pp. I–I.
  * Triggs et al. [2000] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon, “Bundle adjustment—a modern synthesis,” in _Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms Corfu, Greece, September 21–22, 1999 Proceedings_. Springer, 2000, pp. 298–372.
  * Snavely et al. [2006] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: exploring photo collections in 3d,” in _ACM siggraph 2006 papers_ , 2006, pp. 835–846.
  * Beder and Steffen [2006] C. Beder and R. Steffen, “Determining an initial image pair for fixing the scale of a 3d reconstruction from an image sequence,” in _Joint Pattern Recognition Symposium_. Springer, 2006, pp. 657–666.
  * Lepetit et al. [2009] V. Lepetit, F. Moreno-Noguer, and P. Fua, “Epnp: An accurate o(n) solution to the pnp problem,” _International journal of computer vision_ , vol. 81, pp. 155–166, 2009.
  * Hartley and Sturm [1997] R. I. Hartley and P. Sturm, “Triangulation,” _Computer vision and image understanding_ , vol. 68, no. 2, pp. 146–157, 1997.
  * Zhu et al. [2017] S. Zhu, T. Shen, L. Zhou, R. Zhang, T. Fang, and L. Quan, “Accurate, scalable and parallel structure from motion,” Ph.D. dissertation, Hong Kong University of Science and Technology, 2017.
  * Chen et al. [2020] Y. Chen, S. Shen, Y. Chen, and G. Wang, “Graph-based parallel large scale structure from motion,” _Pattern Recognition_ , vol. 107, p. 107537, 2020.
  * Xu et al. [2021] B. Xu, L. Zhang, Y. Liu, H. Ai, B. Wang, Y. Sun, and Z. Fan, “Robust hierarchical structure from motion for large-scale unstructured image sets,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 181, pp. 367–384, 2021.
  * Bhowmick et al. [2017] B. Bhowmick, S. Patra, A. Chatterjee, V. M. Govindu, and S. Banerjee, “Divide and conquer: A hierarchical approach to large-scale structure-from-motion,” _Computer Vision and Image Understanding_ , vol. 157, pp. 190–205, 2017.
  * Snavely et al. [2008] N. Snavely, S. M. Seitz, and R. Szeliski, “Skeletal graphs for efficient structure from motion,” in _2008 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2008, pp. 1–8.
  * Zhou et al. [2017] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1851–1858.
  * Tang and Tan [2018] C. Tang and P. Tan, “Ba-net: Dense bundle adjustment network,” _arXiv preprint arXiv:1806.04807_ , 2018.
  * Wei et al. [2020] X. Wei, Y. Zhang, Z. Li, Y. Fu, and X. Xue, “Deepsfm: Structure from motion via deep bundle adjustment,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_. Springer, 2020, pp. 230–247.
  * Wang et al. [2021] J. Wang, Y. Zhong, Y. Dai, S. Birchfield, K. Zhang, N. Smolyanskiy, and H. Li, “Deep two-view structure-from-motion revisited,” in _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8953–8962.
  * Gu et al. [2021] X. Gu, W. Yuan, Z. Dai, C. Tang, S. Zhu, and P. Tan, “Dro: Deep recurrent optimizer for structure-from-motion,” _arXiv preprint arXiv:2103.13201_ , 2021.
  * Zhuang and Chandraker [2021] B. Zhuang and M. Chandraker, “Fusing the old with the new: Learning relative camera pose with geometry-guided uncertainty,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 32–42.
  * Wu et al. [2022] P. Wu, G. Li, and T. H. Li, “Moac: Multi-level perception optimizer based on dual augmented cost for structure-from-motion,” in _2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval (MIPR)_. IEEE, 2022, pp. 139–145.
  * Moran et al. [2021] D. Moran, H. Koslowsky, Y. Kasten, H. Maron, M. Galun, and R. Basri, “Deep permutation equivariant structure from motion,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5976–5986.
  * He and Habib [2018] F. He and A. Habib, “Three-point-based solution for automated motion parameter estimation of a multi-camera indoor mapping system with planar motion constraint,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 142, pp. 278–291, 2018.
  * Li et al. [2019] J. Li, B. Yang, C. Chen, and A. Habib, “Nrli-uav: Non-rigid registration of sequential raw laser scans and images for low-cost uav lidar point cloud quality improvement,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 158, pp. 123–145, 2019.
  * Lowe [2004] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, pp. 91–110, 2004.
  * Dusmanu et al. [2019] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-net: A trainable cnn for joint detection and description of local features,” _arXiv preprint arXiv:1905.03561_ , 2019.
  * Moulon et al. [2013a] P. Moulon, P. Monasse, and R. Marlet, “Global fusion of relative motions for robust, accurate and scalable structure from motion,” in _Proceedings of the IEEE international conference on computer vision_ , 2013, pp. 3248–3255.
  * Sturm and Triggs [1996] P. Sturm and B. Triggs, “A factorization based algorithm for multi-image projective structure and motion,” in _Computer Vision—ECCV’96: 4th European Conference on Computer Vision Cambridge, UK, April 15–18, 1996 Proceedings Volume II 4_. Springer, 1996, pp. 709–720.
  * Magerand and Del Bue [2017] L. Magerand and A. Del Bue, “Practical projective structure from motion (p2sfm),” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 39–47.
  * Dai et al. [2013] Y. Dai, H. Li, and M. He, “Projective multiview structure and motion from element-wise factorization,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 9, pp. 2238–2251, 2013.
  * Iglesias et al. [2023] J. P. Iglesias, A. Nilsson, and C. Olsson, “expose: Accurate initialization-free projective factorization using exponential regularization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 8959–8968.
  * Hartford et al. [2018] J. Hartford, D. Graham, K. Leyton-Brown, and S. Ravanbakhsh, “Deep models of interactions across sets,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1909–1918.
  * Meier et al. [2012] L. Meier, P. Tanskanen, L. Heng, G. H. Lee, F. Fraundorfer, and M. Pollefeys, “Pixhawk: A micro aerial vehicle design for autonomous flight using onboard computer vision,” _Autonomous Robots_ , vol. 33, pp. 21–39, 2012.
  * Zhu et al. [2018] S. Zhu, R. Zhang, L. Zhou, T. Shen, T. Fang, P. Tan, and L. Quan, “Very large-scale global sfm by distributed motion averaging,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4568–4577.
  * Shi and Malik [2000] J. Shi and J. Malik, “Normalized cuts and image segmentation,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 22, no. 8, pp. 888–905, 2000.
  * Shin and El-Sheimy [2002] E.-H. Shin and N. El-Sheimy, “Accuracy improvement of low cost ins/gps for land applications,” in _Proceedings of the 2002 national technical meeting of the institute of navigation_ , 2002, pp. 146–157.
  * Mildenhall et al. [2021] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021.
  * Ulyanov et al. [2016] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” _arXiv preprint arXiv:1607.08022_ , 2016.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * Moulon et al. [2013b] P. Moulon, P. Monasse, and R. Marlet, “Adaptive structure from motion with a contrario model estimation,” in _Computer Vision–ACCV 2012: 11th Asian Conference on Computer Vision, Daejeon, Korea, November 5-9, 2012, Revised Selected Papers, Part IV 11_. Springer, 2013, pp. 257–270.

## Biography Section

|  Zequan Che received the B.S. degree and M.S. degree in GIS from Wuhan
University, Wuhan, China, in 2020 and 2023, and now he is a researcher in
Guangzhou Urban Planning & Design Survey Research Institute, Guangzhou, China.
His research interests include 3D reconstruction, UAV mapping, point cloud
data processing, and so on.
---|---
|  Jianping Li received the B.S. degree in GIS and the Ph.D. degree in
photogrammetry and remote sensing from Wuhan University, Wuhan, China, in 2015
and 2021, respectively. His research interests include 3D sensing system
integration, UAV/UGV mapping, robot perception, and point cloud data
processing.
---|---
|  Qusheng Li is currently with Guangxi Zhuang Autonomous Region Remote
Sensing Institute of Nature Resources. He mainly engaged in surveying and
mapping, geographical information, and 3D reconstruction. He is a National
Surveying and Mapping Technology Expert.
---|---
|  Bisheng Yang received the B.S. degree in engineering survey, the M.S.
degree, and the Ph.D. degree in photogrammetry and remote sensing from Wuhan
University, China, in 1996, 1999, and 2002, respectively. From 2002 to 2006,
he held a post-doctoral position at the University of Zurich, Switzerland.
Since 2007, he has been a Professor with the State Key Laboratory of
Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS),
Wuhan University, where he is currently the Director of LIESMARS. His main
research interests comprise 3-D geographic information systems, urban
modeling, and digital city. He was a Guest Editor of the ISPRS Journal of
Photogrammetry and Remote Sensing, and Computers & Geosciences.
---|---
|  Zhen Dong received the B.E. and Ph.D. degrees in remote sensing and
photogrammetry from Wuhan University, in 2011 and 2018, respectively. He is
currently a Professor of the State Key Laboratory of Information Engineering
in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University. His
research interest lies in the field of 3D Computer Vision, particularly
including 3D reconstruction, scene understanding, point cloud processing as
well as their applications in intelligent transportation system, digital twin
cities, urban sustainable development and robotics.
---|---