# 3D Reconstruction from Spherical Images: A Review of Techniques,
Applications, and Prospects

San Jiang, Yaxin Li, Duojie Weng, Kan You, Wu Chen S. Jiang and K. You are
with School of Computer Science, China University of Geosciences, Wuhan
430074, China; S. Jiang is also with Department of Land Surveying and Geo-
Informatics, The Hong Kong Polytechnic University, Hong Kong 999077, China.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Li, D. Weng, and W. Chen are
with Department of Land Surveying and Geo-Informatics, The Hong Kong
Polytechnic University, Hong Kong 999077, China. E-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>(Corresponding author: Wu Chen)

###### Abstract

3D reconstruction plays an increasingly important role in modern
photogrammetric systems. Conventional satellite or aerial-based remote sensing
(RS) platforms can provide the necessary data sources for the 3D
reconstruction of large-scale landforms and cities. Even with low-altitude
UAVs (Unmanned Aerial Vehicles), 3D reconstruction in complicated situations,
such as urban canyons and indoor scenes, is challenging due to frequent
tracking failures between camera frames and high data collection costs.
Recently, spherical images have been extensively used due to the capability of
recording surrounding environments from one camera exposure. In contrast to
perspective images with limited FOV (Field of View), spherical images can
cover the whole scene with full horizontal and vertical FOV and facilitate
camera tracking and data acquisition in these complex scenes. With the rapid
evolution and extensive use of professional and consumer-grade spherical
cameras, spherical images show great potential for the 3D modeling of urban
and indoor scenes. Classical 3D reconstruction pipelines, however, cannot be
directly used for spherical images. Besides, there exist few software packages
that are designed for the 3D reconstruction from spherical images. As a
result, this research provides a thorough survey of the state-of-the-art for
3D reconstruction from spherical images in terms of data acquisition, feature
detection and matching, image orientation, and dense matching as well as
presenting promising applications and discussing potential prospects. We
anticipate that this study offers insightful clues to direct future research.

###### Index Terms:

spherical image; equirectangular projection; 3D reconstruction; structure from
motion; simultaneous localization and mapping; dense matching; image matching

##  I Introduction

3D reconstruction is an increasingly critical module in recent photogrammetric
systems. It has been extensively utilized for constructing digital cities[1],
documenting cultural heritages[2], and inspecting tunnel cracks[3], etc. 3D
reconstruction can be implemented by using varying instruments, e.g., LiDAR
(Light Detection and Ranging) scanners, TOF (Time of Flight) sensors, and
optical cameras. The popularity of image sensors and the development of
processing techniques have led to the vast usage of image-based 3D
reconstruction techniques among all available sensors in the field of
photogrammetry and remote sensing (RS), such as satellite and aerial-based
images for urban buildings[4].

Figure 1: 3D reconstruction based on spherical images. (a) a spherical image
and one enlarged region for detail comparison; (b) the sparse reconstruction
model; and (c) the dense reconstruction model after texture mapping.

Due to the relatively high acquisition altitudes and nearly vertical
viewpoints, satellite and aerial-based RS images mainly observe building roofs
over large-scale regions. With the increasing demands for fine-scale modeling,
such as building facades and indoor environments, recent years have witnessed
explosive development of 3D reconstruction based on low-altitude unmanned
aerial vehicles (UAV)[71, 140] or terrestrial mobile mapping systems
(MMS)[158]. Compared with satellite and airborne-based RS platforms, these
near-ground platforms have the advantages of flexible instrument integration
and multi-view imaging, which can record the contents that cannot be observed
from high altitudes. Therefore, UAV and MMS have been used as essential RS
platforms for data acquisitions in urban and indoor scenes[24].

Perspective cameras are the most widely used sensors for image-based 3D
reconstruction. However, due to the characteristics of data acquisition in
street-view and indoor environments, two main issues occur for perspective
cameras with their limited FOV (Field of View) [5]. On the one hand,
surrounding environments are different from aerial photogrammetry. In street-
view and indoor environments, the trajectory of data acquisitions is limited
by street structures and indoor layouts, which causes sudden viewpoint changes
at turning points and track failure between camera frames[6]. On the other
hand, the observation regions are extended from the single-direction records
in aerial photogrammetry to the omnidirectional acquisitions in street-view
and indoor environments. It requires more images at each camera exposure
position and increases acquisition time consumption[7]. Thus, effective
imaging techniques are needed for 3D reconstruction in street-view and indoor
situations.

Spherical cameras, also termed 360 cameras or omnidirectional cameras, can
record all surrounding environments using one camera exposure. In contrast to
traditional perspective cameras, recorded images of spherical cameras can
cover the whole scene, whose FOV ranges are 360 and 180 degrees in horizontal
and vertical directions, respectively. Due to the advantage of spherical
cameras, spherical images have been adopted for 3D modeling in street-view and
indoor environments[8, 9]. In addition, low-cost consumer-grade spherical
cameras like the Insta360 and Ricoh theta[10] are growing in popularity, which
greatly simplifies data acquisition and encourages their use in a variety of
fields, such as damaged building inspection[11], urban environment
analysis[12], urban geo-localization[13, 14], and heritage modeling[15]. Thus,
spherical images have become one of the important data sources for 3D
reconstruction, especially for street-view and indoor environments [16], as
illustrated in Fig. 1.

Spherical images, however, have different characteristics when compared with
traditional perspective images in the context of image-based 3D
reconstruction[17]. One of the most important differences is the camera
imaging model. Consequently, 3D reconstruction from spherical images has
technique differences from perspective images. In addition, fewer commercial
and open-source solutions are designed for the 3D reconstruction from
spherical images when compared with perspective images. Therefore, this study
aims to give a review of reported techniques related to 3D reconstruction from
spherical images. The main contributions of this study include: (1) we give a
systematic and extensive review of recent techniques for 3D reconstruction
from spherical images; (2) we present the most promising applications related
to 3D reconstruction from spherical images; and (3) we also conclude the
prospects for 3D reconstruction from spherical images from the aspects of
technique development and application promotion. The purpose of this study is
to provide useful clues to guide further research for 3D reconstruction from
spherical images.

This paper is organized as follows. The state-of-the-art of data acquisition,
image matching, image orientation, and dense matching for 3D modeling of
spherical images is reviewed in Section II. Section III examines prospective
applications for 3D reconstruction from spherical images, which is followed by
the primary prospects presented in Section IV. Finally, Section V concludes
this work and future studies.

##  II Techniques

This section presents the recent techniques for 3D reconstruction from
spherical images. The main workflow for the photogrammetric 3D reconstruction
is first introduced, followed by data acquisition with varying spherical
cameras, image matching for establishing correspondences, image orientation
for estimating camera poses, and dense matching for producing point clouds.
The details are listed as follows.

###  II-A Main workflow

According to the literature, there are five significant steps in the workflow
of image-based 3D reconstruction, i.e., data acquisition using photogrammetric
systems, image matching to establish correspondences, image orientation to
determine camera poses, dense matching to produce point clouds, and point
cloud meshing and texturing. Spherical images differ in the camera imaging
model and image representation format compared with classical perspective
images, which causes extra considerations in the first four steps. Thus, this
study reviews data acquisition, image matching, image orientation, and dense
matching, and the main workflow of 3D reconstruction for spherical images is
presented in Fig. 2.

Figure 2:  The main workflow of 3D reconstruction for spherical images.

###  II-B Data acquisition

Data acquisition is the first step in the main workflow of 3D reconstruction
for spherical images. In this section, three topics related to data
acquisition are presented, i.e., spherical cameras, image representations, and
acquisition platforms. The details are shown in the following subsections.

####  II-B1  Spherical camera

The development of spherical cameras can be traced back to two centuries
ago[18], which are invented for the documentation of ancient buildings and
cultural heritages. For the purpose of surveying and mapping, spherical
cameras were first used in close-range photogrammetry, in which spherical
images were usually captured by rotating camera around the projection center
or stitching overlapped images through image matching. For aerial
photogrammetry, spherical cameras are designed as an integrated instrument
that consists of several well-calibrated perspective cameras.

Figure 3: The design principle of three commonly used spherical cameras. (a)
dioptric camera; (b) catadioptric camera; and (c) polydioptric camera. TABLE
I: The detailed configurations of well-known spherical cameras. Noticeably,
the resolution for Ladybug5+ is the image dimension of an individual camera.

Camera Manufacturer Principle Number of lenses Resolution(pixel) Weight (kg)
Professional Gear360 Samsung polydioptric 2 4096×2048 0.13 $\times$ Theta Z1
Ricoh polydioptric 2 6720×3360 0.18 $\times$ Theta X Ricoh polydioptric 2
11008×5504 0.17 $\times$ Max 360 GoPro polydioptric 3 5760×2880 0.16 $\times$
Sphere Insta360 polydioptric 2 6080×3040 0.19 $\times$ Pro 2 Insta360
polydioptric 6 7680×7680 1.55 $\checkmark$ Panono Professional360 polydioptric
36 16000×8000 0.48 $\checkmark$ Ladybug5+ Teledyne FLIR polydioptric 6
2048×2464* 3.00 $\checkmark$ Civetta Weiss AG dioptric / 230 M 5.70
$\checkmark$

In recent years, the performance of spherical cameras has been greatly
improved by the progress in the fields of camera sensors and image processing
techniques, e.g., high-resolution digital cameras and high-precision stitching
algorithms. Based on the design principle, spherical cameras can be divided
into three major categories, i.e., dioptric cameras, catadioptric cameras, and
polydioptric cameras[10, 19]:

  * •

Dioptric cameras use a particular lens group to refract rays that compress the
direction of the light entering the subsequent lens group. The obtained FOV
reaches 360 degrees in the horizontal direction and is larger than 90 degrees
in the vertical direction. Thus, two lenses combined back to back can capture
the full surroundings. Fig. 3(a) is an example of the fisheye camera.

  * •

Catadioptric cameras utilize the combination of a lens group for ray
refraction and a special mirror for ray reflection, e.g., a parabolic,
hyperbolic, or elliptical mirror, with a standard camera to achieve the FOV of
360 degrees and greater than 100 degrees in the horizontal and vertical
direction, respectively. Compared with dioptric cameras, this design can
reflect the surrounding light into the subsequent lens. Fig. 3(b) is an
illustration of the catadioptric camera.

  * •

Polydioptric cameras adopt multiple dioptric cameras to obtain a real
spherical FOV, i.e., 360 degrees and 180 degrees in the horizontal and
vertical directions, respectively, in which dioptric cameras have overlapping
FOV to facilitate image stitching. Fig. 3(c) shows an example of a
polydioptric camera comprising 16 cameras.

Among the three categories, polydioptric cameras have become the most
extensively used spherical cameras for both professional and consumer-grade
applications because of two main reasons. On the one hand, it can provide the
full omnidirectional imaging technique; on the other hand, recorded images
have extremely high resolutions due to the use of multiple cameras. Table I
presents the detailed configurations of well-known spherical cameras,
including the consumer-grade cameras, e.g., the Ricoh Theta series and
Insta360 Sphere, and the professional cameras, e.g., Insta360 Pro2 and
Teledyne FLIR Ladybug5+. Except for Weiss AG Civetta, all the other spherical
cameras are designed by using the polydioptric mechanism. For consumer-grade
cameras, the lens number is usually configured as 2 or 3. The illustration of
the cameras is presented in Fig. 4. In general, there are two ways to increase
the resolution of recorded images, i.e., increasing the number of integrated
cameras or exchanging the style of image recording. The former has been used
in Panono and Ladybug5+, which enables instant image recording, e.g., equipped
with an MMS system. The latter has been used in Weiss AG Civetta to obtain
extremely high resolution. This strategy, however, sacrifices the capability
of instant acquisition, and it is more suitable for site-based image
recording.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 4: The illustration of the well-known spherical cameras. (a) Gear360;
(b) Theta Z1; (c) Theta X; (d) Max 360; (e) Sphere; (f) Pro 2; (g) Panono; (h)
Ladybug5+; (i) Civetta.

####  II-B2 Image representation

Spherical images record the surrounding environments at each camera exposure
position. In contrast to the 2D plane representation of perspective images,
Fig. 5 presents the most widely used three types of image representation
methods[20].

  * •

The first one is the spherical representation. Objects in surrounding
environments are mapped onto a sphere, as presented in Fig. 5 (a). Spherical
representation is useful for panoramic navigation, which has been widely used
for street-view navigation, e.g., Google and Baidu Street View. Spherical
representation, however, is unsuitable for image processing and hardware
storage.

  * •

Equirectangular images, which are produced by the equirectangular projection
(ERP), are a common representation, as shown in Fig. 5 (b). Similar to
perspective images, equirectangular images can be considered typical images
and processed by existing algorithms, e.g., feature extraction and
matching[17]. Because of the projection from 3D sphere to 2D plane, geometric
distortions are introduced to equirectangular images, especially for the
regions near sphere poles.

  * •

The third image representation, i.e., cubic-map representation (CMP), is
created to alleviate the distortion in equirectangular projection, which
converts one spherical image into six concentric perspective images, as
presented in Fig. 5 (c). Each cubic-map image can be considered as one typical
perspective image whose projection distortions have been removed. However,
this representation would decrease the overlap region between frames and
increase the image number for image orientation.

Since the simple and typical format, equirectangular representation has been
extensively adopted for spherical images, including well-known open-source and
commercial software packages, e.g., OpenMVG[21], Agisoft Metashape [22], and
Pix4dMapper [23]. Thus, this review pays more attention to the equirectangular
representation of spherical images.

Figure 5: Three typical representation for spherical images: (a) spherical
representation; (b) equirectangular representation; (c) cubic-map
representation.

####  II-B3 Acquisition platform

Depending on their individual properties, spherical cameras can be operated
manually or with various remote sensing platforms. The most extensively
adopted platforms can be moving vehicles[24], ground-fixed tripods[25], and
handheld poles. These platforms record images along urban streets or around
center landmarks, as presented in Fig. 6(a) and Fig. 6(b).

In recent years, UAV platforms are also designed to accommodate spherical
cameras[26], which can record spherical images from relatively high altitudes,
as shown in Fig. 6(c). Compared with spherical cameras for the other two
platforms, e.g., Ladybug5+ for moving vehicles and Civetta for ground-fixed
tripods, spherical cameras for UAVs are strict to the weight and dimension of
sensors due to the limited payload weight and flight endurance. In addition,
spherical cameras are required instant recording ability when mounted on
moving vehicles and UAV platforms, which would restrict the resolution of
recorded images. On the contrary, spherical cameras designed for ground-fixed
tripods can record images with extremely high spatial resolution. Fig. 7 shows
the images recorded by the Weiss AG Civetta spherical camera, from which
details can be observed from both outdoor and indoor recorded spherical
images. In the literature, there exist some useful and public datasets
captured by sphere cameras, which is presented in Table II.

Figure 6: Different acquisition platforms and corresponding sample images. (a)
moving vehicles; (b) ground-fixed tripods; and (c) unmanned aerial vehicles.
Figure 7: Spherical images recorded by the Weiss AG Civetta camera [27]. (a)
the example for the outdoor scene; and (b) the example for the indoor scene.
TABLE II: A list of benchmark datasets for spherical images.

Name Year Purpose Website 3D60 2021 Dense matching
https://vcl3d.github.io/Pano3D It provides composited and realistically
scanned 3D datasets of interior spaces that are used to generate high-quality,
densely annotated spherical panoramas[28]. Re-rendering subsets of Stanford
2D-3D and Matterport 2020 Dense matching https://albert100121.github.io/360SD-
Net-Project-Page/ It contains re-rendered subsets of the Stanford 2D-3D and
Matter-port3D data sets, which consists of 3,577 1024 x 512 stereoscopic ERP
images[29]. Structured3D 2020 Structured 3D modeling
https://structured3d-dataset.org/ It provides 196,515 frames from 3,500
synthetic, photo-realistic house designs, and each scene has 1024 × 512 ERP
images at three different light conditions and camera poses[30]. Re-rendering
subset of Stanford 2D-3D 2019 Dense matching https://github.com/pokonglai/ods-
net It provides a re-rendered subset of the Stanford 2D-3D dataset. It
provides about 50,000 256 × 128 stereo ERP images separated by a 6.5cm
horizontal baseline[31]. PanoSUNCG 2018 Camera motion and depth
https://fuenwang.ml/project/360-depth/ It provides about 25,000 images
captured by SUNCG from 103 different scenes in five camera paths, including
color, depth, and rendering tracks[32]. Stanford 2D-3D 2017 Dense matching
http://3dsemantics.stanford.edu It contains 1,413 full Field of View indoor
realistic capture of six wide range of areas, which provides depth, normal,
and semantic mapping[33]. Matterport3D 2017 Dense matching
https://niessner.github.io/Matterport/ It provides 10,800 CMP panoramas from
90 real building scale scenes, including depth, camera position, and semantic
segmentation[30].

###  II-C Image matching

Image matching is the second step in the main workflow of 3D reconstruction
for spherical images, which aims to establish correspondence matches between
image pairs with high inlier ratio and even spatial distribution. In the
literature, image matching has been a well-studied topic in the fields of
photogrammetry and computer vision, which can be verified from earlier hand-
crafted algorithms to recent learning-based networks[34, 35, 36]. In this
section, we focus on reported methods that can be utilized for spherical
images. Table III lists the algorithms for image matching.

TABLE III: The algorithms for feature matching of spherical images.

Name Language Year Website OmniCV C++ 2020 https://github.com/kaustubh-
sadekar/OmniCV-Lib A computer vision library for omnidirectional cameras,
e.g., dioptric, catadioptric and polydioptric cameras. It provides tools for
format conversion and image viewing. SPHORB C++ 2015
https://github.com/tdsuper/SPHORB A package based on a nearly regular
hexagonal grid parametrization of the sphere geodesic grid, which adapts
planar ORB to the spherical domain[37]. SSIFT Matlab 2009
https://github.com/Artcs1/Keypoints It transforms the plane SIFT to spherical
coordinates and proposes two descriptors for feature matching[38]. Please
refer to the file SSIFT.m in this repository. Tangent Images Python 2020
https://github.com/meder411/Tangent-Images Distortion is alleviated by
rendering the spherical image as a set of local planar image grids tangent to
the subdivided icosahedron[39]. UGSCNN Python 2019
https://github.com/maxjiang93/ugscnn A CNN network for spherical signals based
on parameterized differential operators on unstructured grids[40]. DEEPSPHERE
Python 2020 https://github.com/deepsphere A graphical representation of a
sampling sphere that achieves a controlled balance between efficiency and
rotationally equivalent variance[41]. S2CNN Python 2018
https://github.com/jonkhler/s2cnn A library for the rotation equivariant CNNs
for spherical signals (e.g. omnidirectional images, signals on the globe)[42].
SphereNet Python 2018 https://github.com/ChiWeiHsiao/SphereNet-pytorch A
network that adjusts sampling positions of CNN kernels, which can transfer
existing perspective networks to the omnidirectional case[43].

####  II-C1 Feature detection and description

The purpose of feature detection and description is to detect distinguishable
keypoints that can be found in overlapped images and calculate their robust
descriptors that are invariant to the changes in scale, viewpoint, and
illumination. Due to the equirectangular projection of spherical images, more
serious geometric distortions disturb feature detection and description.
According to the strategy used, existing methods can be divided into four
groups, including classical methods, 2D plane-based methods, 3D sphere-based
methods, and learning-based methods.

(1) Classical methods

Classical feature detection and description methods can also be applied to
equirectangular images because of two main reasons. On the one hand, image
regions near the sphere equator have relatively small distortions after
equirectangular projection; on the other hand, most data acquisition campaigns
are conducted by fixing the roll and pitch angles of sphere cameras, such as
cameras mounted on moving vehicles for street-view images[44] or on fixed
tripods for corridor photos[25]. These two conditions ensure the repeatability
of extracted features on equirectangular images.

In the literature, classical feature detectors have been applied to
equirectangular images, including floating and binary feature descriptors[45,
46, 47, 48]. [44] used SURF (Speeded-up Robust Features) to extract features
from street-view images and conduct image orientation based on SfM (Structure
from Motion). Thus, existing algorithms can be directly utilized for
equirectangular images, which can achieve high efficiency without extra
computational costs. Fig. 8 gives an example of feature extraction using the
classical SIFT.

Figure 8: The influence of geometric distortion on feature matching using the
classical SIFT. The value in each image indicates the rotation angle around
the X axis, and the yellow circles represent the matched feature points
between the two images.

(2) 2D plane-based methods

When spherical images are rotated around the X (pitch angle) or Z (roll angle)
axis, the image appearances near the sphere equator would change obviously,
which causes serious geometric distortions. For visual analysis, Fig. 8
illustrates three images that are captured by rotating around the X axis with
the angle of 0, 45, and 75 degrees, respectively. It is clearly shown that
geometric distortions increase dramatically with the increase of rotation
angles. Because of the introduced distortions, the number of correspondences
decreases in these two corresponding regions. Feature detection and
description must therefore receive more attention. In the field of
photogrammetry, image rectification has been widely used to decrease the
influence of geometric distortions on feature detection and description [49].
Similar to the rectification strategy, some research has been documented for
spherical images, which can be divided into three main groups, i.e., global
methods, semi-global methods, and local methods.

  * •

Global methods aim to rectify the whole image, and cubic-map representation
has been the classical strategy, as presented in Fig. 5(c). For example, [50]
proposed reprojecting spherical images into cubic-map images for feature
detection and implementing a SLAM (Simultaneous Localization and Mapping)
system, termed CubemapSLAM.

  * •

Semi-global methods are designed to reproject a proportion of spherical images
and conduct feature detection on the rectified image region, which is inspired
by the truth that small distortions exist near the sphere equator, as
presented in Fig. 9(b). [51] proposed generating three reprojected
equirectangular images by a rotation of 0, 60, and 120 degrees around X axis,
respectively, and detecting features from the equator regions in each
rectified equirectangular image.

  * •

Local methods aim to rectify local image regions around detected feature
points and calculate their descriptors on the rectified image patches[39, 52],
as presented in Fig. 9(a). In the work of [52], image patches around feature
points are projected to the tangent planes across these feature points, and
SURF descriptors are computed from these patches with similar perspective
views. Although 2D plane-based methods are based on a simple principle, extra
time costs are required in image rectification.

Figure 9: The illustration of 2D plane-based methods. (a) local methods based
on the tangent plane projection of feature point patches; (b) semi-global
methods based on the projection around the sphere equator region. Red shapes
indicate detected features.

(3) 3D sphere-based methods

In contrast to image geometric rectification, some research starts from
scratch to design and implement algorithms for feature detection and
description of spherical images[38, 53, 37]. The primary motivation is to
avoid the high computational costs consumed in image geometric rectification
and decrease the influence of geometric distortions introduced by
equirectangular projection. Naturally, the spherical representation is the
optimal solution instead of the equirectangular representation used in the
above-mentioned methods. Thus, existing research exploits the spherical
coordinate to design new feature detection and description algorithms. The
core of these methods is how to construct the scale space pyramid on spherical
images[54]. Like SIFT, [38] implemented a spherical SIFT, termed SSIFT, which
directly simulates image representation, scale space construction, extreme
point detection, and descriptor calculation in the spherical coordinates, and
shows better performance compared with classical SIFT. Inspired by this work,
others attempt to improve the efficiency of SSIFT by replacing the time-
consuming SIFT with binary detectors and descriptors, such as SPHORB[37] and
BRISKS[53]. Fig. 10 shows feature matching of ORB and SPHORB. In a word, 3D
sphere-based methods can avoid the geometric distortions in equirectangular
images with the sacrifice of efficiency due to the computation on the sphere.

Figure 10: The comparison of feature detection and matching between ORB and
SPHORB. (a) the matching result of ORB; (b) the matching result of SPHORB.

(4) Learning-based methods

Due to the powerful representation learning ability, CNN (Convolutional Neural
Network) based deep learning networks have been extensively used for feature
detection and description. According to the work of [35], existing networks
can be divided into three groups, i.e., joint feature and metric learning
networks[55, 56, 57], separate detector and descriptor learning networks[58,
59, 60],[61], and joint detector and descriptor learning networks[62, 63]. For
the first group, CNN models learn a similarity function to predict image patch
similarities and integrate feature representation and matching within the same
network. For the second group, CNN networks only learn feature representation,
and feature matching is conducted based on classical strategies, such as the
L2-norm Euclidean distance-based metric between feature descriptors. For the
third group, trained models learn detectors and descriptors simultaneously,
which can cope with images recorded in varying conditions. Similar to hand-
crafted methods, existing networks have been used for spherical images by
using pre-trained or finetuned models, as shown in Fig. 11(a). For further
details, readers can refer to[64],[65].

Due to inevitable distortions in sphere-to-plane projection, CNN networks for
perspective images may obtain inaccurate results. To address this issue,
reported approaches in literature can be divided into three groups, i.e.,
tangent projection methods, CNN kernel shape resizing methods, and CNN
sampling point adjustment methods.

Figure 11: The principle of CNN for spherical images[66],[67]. (a) direct
applying CNN on the ERP spherical images; (b) applying CNN on the tangent
images; (c) adjusting the kernel shape of CNN at varying latitudes; (d)
adjusting the sampling points of CNN kernels by using tangent plane
projection.

  * •

For the first one, similar to 2D plane-based methods, equirectangular images
are first projected to undistorted tangent images[39] or divided into quasi-
uniform discrete images[68], and then existing CNN networks are applied to
resulting images, as shown in Fig. 11(b). Although these methods can achieve
high accurate predictions, they suffer from high computational costs due to
resampling of 3D spherical images to 2D plane images.

  * •

For the second one, CNN networks are designed to work on equirectangular
images by adjusting the CNN kernel shape[43],[66],[67]. In the work of [66], a
network termed SPHCONV has been proposed, which aims to produce results as the
output of applying perspective CNN networks to the corresponding tangent
images. SPHCONV was achieved by defining convolution kernels with varying
shapes for pixels in different image rows, as illustrated in Fig. 11(c).
Similarly, [69] proposed a kernel transformer network (KTN) to learn spherical
kernels by taking as input the latitude angle and source kernels for
perspective images.

  * •

For the third one, sampling points of CNN kernels are adjusted based on ERP
sphere image distortions instead of adjusting convolution kernel shape. For
example, [67] and [43] designed distortion-aware networks that sample non-
regular grid locations according to the distortions of different pixels, as
shown in Fig. 11(d). The core idea is to determine sampling locations based on
the sphere projection of a regular grid on the corresponding tangent plane.
Due to regular convolution kernels, these frameworks enable the transfer
between CNN models for perspective and equirectangular images.

####  II-C2 Feature matching

The purpose of feature matching is to search correspondences from two sets of
feature descriptors. Generally, feature matching is achieved by using nearest-
neighbor searching based on the L2-norm Euclidean distance metric[70]. In the
literature, extensive research has been reported for feature matching from the
aspects of efficiency acceleration and precision improvement[34],[71]. Due to
the high dimension of local feature descriptors and the large number of
spherical images, exhaustive feature matching consumes extremely high time
costs. Thus, it becomes very critical to increase the efficiency of feature
matching.

In the literature, there are valuable methods for accelerating feature
matching, including restricting the number of features in feature matching,
decreasing the number of images, and reducing the number of match pairs. For a
detailed review, the readers can refer to this work[34],[72]. Among these
reported methods, match pair selection can be the most straightforward and
effective way to accelerate feature matching. Three strategies can be
exploited for spherical images based on their acquisition environments. First,
data acquisition constraints can be used, e.g., the time-sequential constraint
for street-view images. In other words, feature matching can be restricted to
neighbors according to their acquisition time. Second, for professional MMS
systems and consumer-grade sensors, precise or rough GNSS (Global Navigation
Satellite System) data is usually recorded simultaneously with spherical
images, which provides the best clue to select spatially overlapped match
pairs. Third, the CBIR (Content-based Image Retrieval) technique has become a
standard module[73],[74]. It merely uses images for visual retrieval without
other assumptions and auxiliary data. Fig. 12 shows an example of CBIR.

Figure 12: An example of CBIR for spherical images. The image with blue
borders is the query image, and the other images are the retrieval results.
Images with green and red borders represent true and false retrieval results,
respectively.

###  II-D Image orientation

Image orientation aims to estimate camera poses, scene structures, and
intrinsic parameters based on established two-view correspondences. Image
orientation is also termed aerial triangulation (AT) in photogrammetric 3D
reconstruction, which requires good initial values of unknown parameters and
well-designed configurations in data acquisition. Image orientation of
spherical images can be achieved through the well-known Structure from Motion
(SfM) technique for multi-view geometry in computer vision[75] or the
Simultaneous Localization and Mapping (SLAM) technique for instant
localization in robot vision[76]. Thus, this subsection presents camera
calibration, SfM-based offline, and SLAM-based online methods for spherical
image orientation.

####  II-D1 Camera calibration

(1) camera imaging model

The camera imaging model is the basis for camera calibration, which
establishes the geometric transformation between 3D points in the object space
and 2D points in the image plane. Based on the design of spherical cameras,
there are three major camera imaging models, i.e., the unified projection
model, general camera model, and multi-camera model.

  * •

Unified projection model[77] has been mainly designed for central catadioptric
cameras, in which environment light rays intersect in a single point, i.e.,
the projection center of the mirror, as shown in Fig. 13(a). This camera
imaging model follows a strict theoretical projection function that models
real-world imaging errors. The unified projection model has recently been
verified as effective for wide-angle and fisheye cameras[78].

  * •

General camera model (Taylor model)[79] has been used for modeling the imaging
procedure of central catadioptric and dioptric cameras. Instead of using the
strict theoretical model in the unified projection model, the general camera
model utilizes a Taylor polynomial function to fit the projection.

  * •

Multi-camera model has been designed to establish the projection of the widely
used polydioptric cameras that record spherical images by using a camera rig,
e.g., the Ladybug 5+ camera. The multi-camera model can be implemented using
several individual camera models or a unit sphere camera model[80]. The former
is more rigorous in formulating the imaging system, as shown in Fig. 13(b); on
the contrary, the latter has a more straightforward formula widely used in
close-range photogrammetry, as shown in Fig. 13(c).

Figure 13: Camera imaging models[80],[79]. (a) central catadioptric cameras;
(b) multi-camera model; and (c) unit sphere camera model.

(2) camera calibration

The purpose of camera calibration is to calculate the intrinsic parameters of
cameras, e.g., the focal length, principle point, and lens distortion
coefficients. After selecting a proper camera imaging model, camera
calibration can be achieved by using a combined bundle adjustment of both
interior and exterior orientation parameters. In practice, there are three
ways to estimate the parameters: self-calibration based on the epipolar
geometry constraint, space resection using non-planar 3D control points, and
laboratory calibration using checkerboards[81]. In the work of [82], a Ricoh
Theta dual-camera system has been calibrated based on the expanded unit sphere
camera model, in which two extra parameters that model the displacement of
latitude and longitude coordinates are incorporated into the camera imaging
model. For calibrating a professional Ladybug multi-camera system, [83]
adopted the colinear equation-based rigorous model, which combines five radial
distortion parameters and two decentering distortion parameters to model the
imaging errors. For comparing different calibration methods for spherical
cameras, readers can refer to the work of [84]. Besides, Table IV presents a
list of open-source software packages for spherical camera calibration.

TABLE IV: A list of open-source software packages for spherical camera
calibration.

Name Language Year Website OCamCalib Matlab 2006
https://sites.google.com/site/scarabotix/ocamcalib-omnidirectional-camera-
calibration-toolbox-for-matlab A toolbox to calibrate any central
omnidirectional camera, i.e., panoramic cameras having a single effective
viewpoint[79]. Improved OcamCalib Matlab 2015
https://github.com/urbste/ImprovedOcamCalib An add-on toolkit to the OCamCalib
toolbox that implements calibration algorithms for wide-angle, fisheye, and
omnidirectional cameras[81]. LIBOMNICAL Matlab 2014
https://www.cvlibs.net/projects/omnicam a MATLAB Toolbox to calibrate central
and slightly non-central catadioptric cameras and catadioptric stereo
setups[85]. Omnidirectional Calibration Toolbox Mei Matlab 2007
https://www.robots.ox.ac.uk/~cmei/Toolbox.html A toolbox implements the
unified projection model to calibrate hyperbolic, parabolic, and folding
mirrors and spherical and wide-angle sensors[77]. camodocal C++ 2013
https://github.com/hengli/camodocal A toolbox for automatic intrinsic and
extrinsic calibration of a camera rig with multiple generic cameras and
odometry[78]. kalibr C++ 2016 https://github.com/ethz-asl/kalibr A toolbox for
multi-camera calibration and multi-sensor integration[86].

####  II-D2 SfM-based offline methods

(1) Principle of incremental SfM

Existing SfM can be divided into three major groups, i.e., incremental SfM,
global SfM, and hybrid SfM, according to the used strategy for estimating and
optimizing unknown parameters[87]. Compared with other techniques, incremental
SfM has the advantages of resisting high outlier ratios and achieving accurate
orientation models. Incremental SfM has been widely used in photogrammetric 3D
reconstruction[49].

The workflow of incremental SfM is shown in Fig. 14, which consists of two
components, i.e., feature matching and incremental image registration. Feature
matching can be solved by using the methods presented in Section III-A. For
image registration, consistent correspondences are first tied to create tie-
points[88]. In incremental image registration, two seed images are first
selected among all matched image pairs, which have a large enough intersection
angle and a sufficient number of well-distributed matched features. A base
model is then constructed by recovering their relative poses and triangulating
3D scene points, which would be used to register the next-best image and
triangulate more 3D scene points. Meanwhile, local or global bundle adjustment
(BA) optimization is executed to decrease accumulated errors and remove false
matches. After iterative image registration and point triangulation, all
images are registered into the same 3D model[89].

Figure 14: The workflow of the incremental SfM[72].

In the above-mentioned iterative local and global BA, the optimization problem
is usually formulated as a joint minimization of the reprojection function
[90], where the sum of errors between tie-point projections and their
corresponding image points is minimized. The object function of BA is
presented by the Equation (1).

$\min\limits_{C_{j},X_{i}}\sum_{i=1}^{n}\sum_{j=1}^{m}\rho_{ij}\parallel
P(C_{j},X_{i})-x_{ij}\parallel^{2}$ (1)

where $X_{i}$ and $C_{j}$ indicate a 3D point and a camera, respectively;
$P(C_{j},X_{i})$ is the projection of point $X_{i}$ on camera $C_{j}$;
$x_{ij}$ is an observed image point; $\left\|\bullet\right\|$ denotes L2-norm;
$\rho_{ij}$ is an indicator function with $\rho_{ij}=1$ if point $X_{i}$ is
visible in camera $C_{j}$; otherwise $\rho_{ij}=0$.

(2) Perspective SfM to spherical SfM

For spherical image orientation, there have been some attempts to adapt
perspective SfM to spherical SfM in the last two decades. [91] proposed a two-
step SfM for omnidirectional images depending on linear initialization and
nonlinear optimization, which only recovers camera relative geometry.
Uncertainty analysis and result comparison were also conducted and compared
with perspective SfM. In the work of [92], both two-view and three-view
geometry of spherical images have been analyzed and discussed. The epipolar
geometry forms the basis for 3D reconstruction from spherical images, e.g.,
the pose recovery in a virtual navigation system[93].

With the usage of advanced MMS systems for urban city modeling and navigation,
some researchers moved their attention to 3D reconstruction of large-scale
scenes instead of two or three-view geometry estimation in the earlier
work[94]. As one of the early pioneering attempts, [44] proposed an
incremental SfM system that combines state-of-the-art techniques, e.g., local
feature-based feature detection, approximate nearest neighbor-based feature
matching, and robust essential matrix estimation and optimization. The
performance of the proposed SfM system has been verified by using Google
Street View images that cover a large street block[24], as presented in Fig.
15(a). For full spherical images, [17] investigated different error metrics on
relative and absolute pose estimation and also designed the error
approximations to reduce computational costs. These error metrics consist of
the basic blocks for spherical SfM.

In contrast to professional sensors, recent years also witnessed the explosive
development of consumer-grade spherical cameras[10] and their usage in 3D
modeling. In [95], the von Mises-Fisher distribution was utilized to model the
noise distribution of feature point positions on spherical images and to
reformulate the error function in the bundle adjustment optimization.
Meanwhile, spherical-n-point and triangulation algorithms that are suitable
for spherical images have been proposed to achieve spherical video orientation
and viewing direction stabilization. [26] embedded spherical cameras into a
UAV platform and proposed a solution for panoramic photogrammetry, as
presented in Fig. 15(b). In data processing, original spherical images are
first reprojected to cubic images, which are then reconstructed by using
existing commercial software. The study verifies the validation of spherical
cameras for oblique photogrammetric 3D modeling of urban buildings.

Figure 15: SfM for spherical images. (a) direct processing of spherical images
based on the unit spherical camera model[44]; (b) indirect processing of
spherical images based on sphere-to-plane projection[26].

Although extensive research has been reported in the literature, the majority
of both open-source and commercial software packages are now designed for
perspective images. Table V lists the well-known and widely used software
packages for 3D reconstruction in the fields of photogrammetry and computer
vision. It is shown that only a few packages provide the full 3D
reconstruction module for spherical images, e.g., the open-source software
OpenMVG and MicMac and the commercial software Pix4Dmapper and Metashape. With
the increasing popularity of spherical cameras, there is an urgent requirement
for well-designed toolkits that can support scientific research and
engineering application. Thus, further research is required for
photogrammetric 3D reconstruction from spherical images.

TABLE V: A list of open-source and commercial SfM software packages.

Name Language Year Website Spherical camera AliceVision C++ 2018
https://github.com/alicevision $\times$ A photogrammetric computer vision
framework for 3D reconstruction. It provides three types of camera models,
i.e., Radial, Brown, and Fisheye[96]. COLMAP C++ 2016
https://github.com/colmap/colmap $\times$ A 3D reconstruction software that
includes both sparse and dense matching modules, which includes Radial, and
Fisheye camera models[97]. OpenMVG C++ 2015 https://github.com/openMVG/openMVG
$\checkmark$ A well-known sparse reconstruction software that provides Radial,
Brown, Fisheye, and Sphere camera models. It can directly process spherical
images[21]. MicMac C++ 2007 https://github.com/micmacIGN/micmac $\checkmark$
An open-source photogrammetric toolkit that provides modules of AT, dense
matching, and ortho-rectification for satellite, aerial and close-range
images[98]. ContextCapture / 2022 https://www.bentley.com $\times$ A well-
known and widely used 3D modeling software in the field of photogrammetry,
which can provide the best 3D textured mesh models. It only supports
perspective cameras. Pix4Dmapper / 2022 https://www.pix4d.com $\checkmark$ A
well-known and widely used photogrammetric software for aerial and close-range
images in the field of photogrammetry. It supports perspective, fisheye, and
spherical cameras. Metashape / 2022 https://www.agisoft.com $\checkmark$ A
well-known and widely used photogrammetric software for aerial and close-range
images in the field of photogrammetry. It supports perspective, fisheye, and
spherical cameras. RealityCapture / 2022 https://www.capturingreality.com
$\times$ A photogrammetric computer vision software for 3D reconstruction. It
features a fast speed for image processing. It now supports a perspective
camera.

TABLE VI: A list of open-source SLAM software packages.

Name Language Year Website Spherical camera ORB-SLAM3 C++ 2021
https://github.com/UZ-SLAMLab/ORB_SLAM3 $\times$ A SLAM system that supports
perspective and fisheye cameras and can be embedded with monocular, stereo,
and RGBD cameras[99]. OpenVSLAM C++ 2019
https://github.com/xdspacelab/openvslam $\checkmark$ A visual SLAM system that
supports various camera models, including perspective, fisheye, and spherical
cameras[100]. Cubemap-SLAM C++ 2018 https://github.com/nkwangyh/CubemapSLAM
$\times$ A visual SLAM system that converts fisheye images into perspective
images and achieves image orientation based on the ORB-SLAM[50]. DSO C++ 2017
https://github.com/JakobEngel/dso $\times$ A direct sparse visual odometry
that supports only perspective cameras[101]. MultiCol-SLAM C++ 2016
https://github.com/urbste/MultiCol-SLAM $\times$ A multi-fisheye SLAM that
supports rigidly coupled multi-camera systems. It extends the ORB-SLAM and
ORB-SLAM2 systems[102]. LSD-SLAM C++ 2014 https://github.com/tum-
vision/lsd_slam $\times$ A direct monocular SLAM system that enables real-time
image orientation and semi-dense depth map generation[103].

####  II-D3 SLAM-based online methods

(1) Principle of visual SLAM

In contrast to the offline SfM technique, SLAM can implement simultaneous and
real-time image orientation and scene reconstruction, which origins from the
robotic field for localization and navigation without GNSS signals, as well as
environment mapping[104]. In the field of photogrammetry and computer vision,
SLAM has also been used for online image orientation, e.g., UAV and MMS
images[105],[106]. Various sensors can be integrated into a SLAM system, such
as RGB and depth cameras, laser scanners, GNSS and IMU (Inertial Measurement
Unit) instruments, and wheel odometry[104]. For spherical image orientation,
the used SLAM system is termed visual SLAM.

Fig. 16 shows the workflow of the visual SLAM, which consists of two major
components. The front-end is utilized to process sequentially observed images
through feature extraction, feature matching, motion estimation, and keyframe
selection; the back-end is responsible for loop detection, BA optimization,
and 3D mapping. The processing pipeline of the visual SLAM system includes: 1)
sequentially estimating the motion of newly added images with local BA
optimization; 2) checking whether or not the newly added images are keyframes
and creating more map points from keyframes; 3) detecting loops between newly
added images and existing 3D models and conducting global BA optimization to
reduce error accumulation. Through the iterative execution of these three
steps, images are sequentially oriented.

Figure 16: The workflow of the visual SLAM system.

(2) SLAM with large FOV cameras

Cameras with large field-of-view angles can enhance the robustness of the SLAM
systems[7] since they enable long-period feature tracking, especially in urban
streets and indoor environments. Large FOV cameras have been increasingly
integrated into visual SLAM systems. Recently adopted large FOV cameras can be
grouped into two categories, including fisheye cameras with wide FOV and
spherical cameras with full FOV.

Fisheye cameras are the most reported large FOV sensors coupled with visual
SLAM systems[107, 108, 109, 7]. [107] proposed a direct monocular SLAM based
on a unified omnidirectional camera model. It was an extension of their
previous LSD-SLAM[103], and it has superior performance on the accuracy of
localization and robustness to strong rotational movement. In the work of
[108], the authors also designed a direct monocular SLAM system, which is an
extension of the DSO system[101] and integrates the unified camera model
proposed in [107]. [50] proposed a feature-based SLAM system, termed Cubemap-
SLAM, based on the cubic projection of fisheye images. In this work, the
cubemap model has been embedded into the ORB-SLAM. To evaluate the impact of
large FOV cameras, [7] conducted a series of tests by using both indoor and
outdoor datasets, and their evaluation reveals that the reconstruction
accuracy also depends on the environment, and large FOV cameras tend to
improve performance in indoor scenes.

Spherical cameras can further extend the FOV of fisheye cameras, and they have
been widely used for data acquisition in MMS systems[24, 44]. [110] proposed a
feature-based SLAM workflow that only estimates the pose of the latest image
without the execution of BA optimization. The core idea is to decouple the
problem of pose recovery into rotation estimation from epipolar geometry and
translation estimation using 3D-2D correspondences. [102] extended the ORB-
SLAM[111, 112] system into a multi-fisheye omnidirectional SLAM system, termed
MultiCol-SLAM, which supports the arbitrary rigidly coupled multi-camera
systems. In [6], a fisheye calibration model has been designed and used for
multiple fisheye camera rigs, as shown in Fig. 17(a), and a feature-based
SLAM, termed PAN-SLAM, has been proposed based on well-designed strategies for
initialization, feature matching and tracking, and loop detection. The tests
demonstrate the validation of the proposed SLAM system.

Figure 17: The visual SLAM systems based on (a) the multi-fisheye camera
rig[6] and (b) the full spherical camera[100].

In recent years, spherical images captured by dual-fisheye cameras are
becoming more and more popular, which promotes the development of panoramic
SLAM systems[113, 114, 100, 115]. [100] designed a versatile visual SLAM,
termed OpenVSLAM, which implements perspective, fisheye, and spherical camera
models and supports monocular, stereo, and RGB-D cameras, as shown in Fig.
17(b). In addition, the authors provided an indoor benchmark for performance
evaluation[116]. [115] investigated varying feature detection and description
techniques, including SPHORB and SSIFT, and compared their performance in
spherical image orientation based on SLAM. Recently, [113] implemented a
direct SLAM system that extends DSO (Direct Sparse Odometry) with the
spherical camera model to process equirectangular images. Table VI presents a
list of open-source SLAM software packages. The same finding can be made as
for SfM-based methods that few SLAM systems support full spherical cameras.

###  II-E Dense matching

Dense matching aims to establish pixel-wise correspondences and generate point
clouds from SfM or SLAM-based oriented images. According to the 3D
reconstruction pipeline shown in Fig. 2, generated dense point clouds would be
used to construct detailed 3D models after point meshing and texture mapping.
Thus, the performance of dense matching methods would determine the precision
and completeness of the final 3D models. In the literature, dense matching has
been an extensively studied topic with the arising of 3D reconstruction in
photogrammetry and computer vision. In addition to traditional methods for
perspective images, recent years have witnessed increasingly reported dense
matching methods for spherical images. Generally, this work can be grouped
into single-view depth prediction and multi-view stereo matching.

####  II-E1 Single-view depth prediction

Single-view depth prediction methods can process an individual spherical image
instead of pixel-wise correspondence searching between images. In the
literature, single-view depth prediction methods are usually implemented
through learning-based CNN networks[117, 118, 119, 120, 28], which belong to a
new research topic in the last five years. As one of the earliest works, [28]
prepared and released a spherical training dataset with ground-truth depth
maps, as presented in Fig. 18(a). It can be used as the training dataset for
depth estimation networks for spherical images instead of training on
perspective datasets with sub-optimal performance. Considering the correlation
between depth and geometric structure in indoor environments, [119] attempted
to leverage existing geometric structures, e.g., corners, boundaries, and
planes, as priors for network training or inferring these structures from
generated depth estimation. The reconstruction models are presented in Fig.
18(b). For indoor scenes, [120] suggested representing equirectangular images
by using vertical slices of the sphere, which partitions input images into
vertical slices. Based on the LSTM (long short-term memory), the authors
designed a network called SliceNet for depth estimation.

In contrast to indoor scenes, [117] prepared the Depth360 dataset and designed
an end-to-end two-branch network, termed SegFuse, for depth prediction of
spherical images. The core idea is to integrate the segmentation of cubic-map
images from one branch into the depth estimation of equirectangular images
from the other branch.

Figure 18: Single view depth prediction for spherical images. (a) a prepared
360 dataset[28]; (b) indoor depth prediction from spherical images[119].
Figure 19: multi-view dense matching for spherical images. (a) revised PMVS
algorithm[121]; (b) sphere sweeping algorithm[114]; and (c) the finetuned CNN
network[118].

####  II-E2 Multi-view stereo matching

(1) existing method

Multi-view stereo matching methods attempt to recover dense point clouds from
two or multiple oriented images based on SfM and SLAM image orientation
techniques[122]. Existing methods can be divided into two groups based on the
strategies used. For the first one, traditional hand-crafted methods are on
the one hand revised to adapt to spherical images. In the work of[17, 121],
the PMVS (Patch-based Multi-view Stereo) library was revised to directly
process spherical images for dense matching, and the results are shown in Fig.
19(a). For the second one, spherical images are projected into cubic-map
images, and traditional hand-crafted algorithms and deep learning-based
networks are adopted straightforward since they are designed or trained for
perspective images, e.g., hand-crafted methods[123, 124, 125, 126] and
learning-based methods[127, 128, 129, 130, 131]. These two strategies can make
use of existing well-designed methods.

(2) redesigned method

For this category, algorithms are designed and implemented from scratch
considering the characteristic of spherical images[5, 114, 132, 133, 29]. In
[132], a full workflow was designed for 3D reconstruction of spherical image
pairs, in which a stereo matching algorithm was proposed based on a partial
differential equation (PDE), and the complete 3D scene was obtained by
registering partial models. Considering the power of the plane sweeping
algorithm, [114] proposed a sphere sweeping algorithm based on the unified
omnidirectional camera model, in which virtual spheres were used instead of
virtual planes. Matching results are shown in Fig. 19(b). The algorithm was
tested by using both synthetic and real datasets. These methods can be seen as
mimics of the traditional dense matching algorithm for perspective images.

Recently, CNN networks have also been adopted to design multi-view stereo
solutions. In the work of [29], a dual-camera imaging system that consists of
top and bottom cameras was designed, which ensures that the epipolar lines of
captured images are vertically aligned. A polar angle layer was added to a
two-branch network for depth estimation, which plays as additional input
geometric information to supervise model training. [118] proposed a complete
solution for 3D reconstruction from spherical images, including image
orientation, epipolar rectification, dense matching, and texture mapping. For
dense matching, the authors adopted an existing network that was retrained
using a synthetic dataset to learn spherical disparity and designed a
spherical binoctree structure for depth map fusion. The proposed solution
achieves superior performance when compared with traditional methods, as
presented in Fig. 19(c). In conclusion, dense matching methods based on CNN
networks attract attention in recent years. This review would not cover all
aspects in this field.

##  III Applications

This section presents promising applications related to spherical images. Due
to the characteristics of full FOV and low cost, spherical images have been
used widely in a variety of applications. According to the purpose of this
study, the reviewed applications would be restricted to 3D reconstruction,
including cultural heritage documentation, urban modeling and navigation,
tunnel mapping and inspection, and other applications, e.g., urban tree
localization, emergency response and rescue, and underwater occlusion
avoidance.

###  III-A Cultural heritage documentation

Cultural heritage documentation (CHD) is the earliest usage of spherical
images, which can be dated back to two centuries ago[18]. In contrast to the
large-scale acquisition required in aerial photogrammetry, the scale of
cultural heritage documentation is much smaller but with serious occlusions,
e.g., building pillars and inner structures. It requires close-range data
acquisition with a large FOV to decrease human labor and ensure completeness.
When compared to conventional aerial plane-based and recent UAV-based
photogrammetry, spherical images can be recorded in a more flexible way, e.g.,
ground-fixed tripods and hand-held poles, and have been widely used for
cultural heritage documentation.

In the field of photogrammetry and remote sensing,[134, 135, 15] have made an
earlier contribution to CHD by using spherical images. This work presents the
basic principle of the proposed spherical photogrammetry (SP), i.e., the
collinear equation of spherical imaging and the coplanarity condition for
image orientation. By using two church architectures, this study verifies the
advantages of spherical images with the remarkable efficiency of data
acquisition, the completeness of documentation, and the low economic cost
compared with other photogrammetric instruments.

With the explosive development of consumer-grade spherical cameras, recent
researchers have also turned to adopting low-cost cameras for data acquisition
and 3D reconstruction of CHD. For example, [9] used a Panono 360 camera for
the image collection of two churches, which is a ball-shaped camera that
consists of 36 camera submodules to record surrounding environments. This
study verified the centimeter-level precision of dense point clouds from SfM
and MVS-based photogrammetric processing. [136] exploited a Xiaomi Mi Sphere
360 camera for the precision assessment of 3D modeling. In this study, a Leica
TS30 total station has been used for an in-site survey of ground control
points (GCPs), and two software packages, including Agisoft Metashape and
Pix4dMapper, have been evaluated. Experiments demonstrate that millimeter-
level accuracy has been obtained in both image orientation and 3D
reconstruction by using GCPs and laser scanning point clouds.

Because of the complex structure of cultural heritages, aerial and ground
images have also been combined to reconstruct complex cultural heritages. [25]
combined low-altitude UAV images and hand-held spherical images for 3D
documentation of a bell tower. In this work, a DJI Phantom 4 Pro UAV has been
adopted for scanning external structures under a properly designed flight
trajectory, and two spherical cameras, i.e., the GoPro Fusion and the Kandao
Qoocam 8K, have been used for scanning internal structures. Based on an SfM
and MVS-based processing solution, the reconstructed 3D model has high-quality
details in the external and internal of the complex bell tower. In contrast to
using consumer-grade spherical cameras, [22] conducted high-quality digital
twin documentation of a well-known town, Long Hu Gu Zhai at Guangdong
Province, China. This study uses professional Weiss AG Civetta spherical
cameras for terrestrial images with 230 megapixels resolution and a UAV for
along-street aerial images. By using the Agisoft Metashape, 3D models were
reconstructed from aerial and terrestrial images, as presented in Fig. 20. The
results demonstrate the potential of aerial and ground images for the
precision documentation of complex buildings.

Figure 20: Cultural heritage documentation by using aerial UAV images and
terrestrial spherical images[22]. (a) Data acquisition based on the Weiss AG
Civetta camera; (b) image orientation of aerial-ground images; (c) dense
matching of building inner structures; (d) 3D reconstruction of buildings.

###  III-B Urban modeling and navigation

Spherical cameras have been integrated with mobile mapping systems (MMS) for
urban street modeling and navigation. The most famous application of spherical
images comes from Google Street View, which provides immersive navigation and
observation along urban streets. At the beginning of Google Street View,
collected street images have also been used for 3D modeling of building
facades in addition to image navigation[24],[8], as shown in Fig. 21(a). [137]
designed a unified framework to reconstruct 3D models from spherical images
captured from a Ladybug camera, in which bundle adjustment free image
orientation, piecewise planarity constrained dense matching, and novel depth
map fusion were reported considering the characteristics of low-texture,
repetitive pattern, and large light changes in urban streets.

In contrast to mesh-based 3D models, recent studies show well-formed wire-
frame models reconstructed from spherical images. Under the Manhattan world
assumption, [132] converted spherical images into cubic-map representation and
segmented urban scenes into planar structures under the constraints of image
color, and edge and normal information from MVS-based depth maps, as presented
in Fig. 21(b). By using street images and deep learning techniques, [138]
designed a workflow for building detection and height calculation and created
3D models for large-scale urban scenes.

Due to the high image overlap of large FOV cameras, spherical images have also
been widely used to achieve localization and navigation in complex urban
environments. [13] proposed a solution for the large-scale localization of
photos without geotagged labels. In this study, 3D sparse models reconstructed
from street view images were used as the reference data source, and a three-
step algorithm was designed for geo-localization, which includes image
retrieval-based coarse geo-localization, reliable feature matching between the
query image and retrieved candidate images, and the PnP (Perspective n Points)
based precise geo-localization. Instead of using low-level features, [139]
turned to detect high-level semantic information from spherical images, such
as lamp posts and street signs, which are detected based on a retrained YOLO
CNN network. To geo-localizing interesting targets within urban scenes to
assist vehicle navigation, [140] proposed a line of bearing (LOB) based
positioning method for urban street objects, e.g., road lamps, shown in Fig.
21(c).

Figure 21: Urban modeling and navigation. (a) mesh models from Google Street
View[24]; (b) 3D models from spherical images[132]; (c) street object
localization[140].

###  III-C Tunnel mapping and inspection

Artificial tunnels are extensively used in daily life, e.g., traffic tunnels
and drainage pipes. Regular inspection is essential to ensure normal functions
and extend the lifespans. In practice, the inspection work has been widely
achieved through labor-sensitive operators. In recent years, with the advent
of small-size spherical cameras and the development of image processing
techniques, image-based inspection solutions have also been presented and
verified to achieve automatic tunnel mapping and inspection[141].

In the work of [142], a low-cost and flexible system has been designed to
create a panorama image of the traffic tunnels, in which SfM-based image
orientation and stitching-based image registration techniques have been used
under the constraint of tunnel design data. This work verified the feasibility
of image-based mapping for tunnel in-field inspection. Based on the rapid data
acquisition ability of spherical cameras, [143] designed an overall framework
for photogrammetric tunnel mapping. In this work, an Insta360 Pro sphere
camera integrating six fisheye camera modules has been used to record images
with 4000 by 3000 pixels. During data acquisition, the spherical camera was
installed on a tripod to image recording with long exposure times. For quality
verification, point clouds were also collected by using a Riegl VZ-400i TLS
scanner. Experimental results demonstrate that the reconstructed tunnel models
can achieve millimeter level precision when compared with TLS point clouds, as
presented in Fig. 22(a).

In contrast to large-size tunnel mapping, small-size gas and drainage pipes
are the other typical inspection scenes that require the full FOV acquisition
ability of a spherical camera due to the limited space in these pipes. In the
work of[144], drainage pipe inspection has been implemented by using low-cost
GoPro Fusion video cameras. Key frames were extracted from recorded videos and
converted to cubic-map images, then used to reconstruct 3D models based on
typical SfM and MVS-based solutions. For the inspection of drainage pipes with
minimal space, [145] designed a pipeline capsule machine (PCM), as presented
in Fig. 22(b), which mainly consists of a large FOV camera, a LED (light-
emitting diode) lighting instrument, and a power supply module. The PCM sensor
can record images with the flow of water, which are further used for crack and
erosion detection based on deep learning-based methods[146].

Similarly, for the inspection of gas pipes,[147] designed an inspection robot
that is equipped with a catadioptric camera for spherical image collection.
Unlike the PCM, depending on flowing water, the inspection robot can operate
on other pipes since it uses wheels for forward moving. In a word, spherical
cameras have good advantages for tunnel mapping and inspection due to the low
cost of sensors and the limited space of tunnels.

Figure 22: Tunnel mapping and inspection. (a) mine tunnel mapping[143]; (b)
drainage pipe inspection[146].

###  III-D Other applications

In recent years, spherical images have also been used in other applications,
e.g., emergency response and rescue, urban tree detection and mapping,
collision avoidance of underwater vehicles, etc. [11] designed an integrated
photogrammetric system for rapid data acquisition and damaged building
inspection. This work took advantage of low-altitude UAVs and ground backpack
MMSs to collect aerial-ground images and validate the integration of aerial-
ground images for the full 3D reconstruction of urban buildings to boost the
observation completeness.

Tree number, location, and structure are critical for the precision management
of vegetation resources. Although conventional aerial photogrammetry and LiDAR
(Light Detection and Ranging) can provide large-scale RS data, they suffer
serious occlusions of urban scenes and cannot achieve accurate measurements of
street trees. Considering these issues, [148] proposed using spherical cameras
for the 3D modeling and parameter calculation of urban trees. In this study,
3D models were reconstructed based on SfM, which were further processed to
extract individual trees, as shown in Fig. 23(a). Based on the 3D models, some
useful parameters, such as trunk diameter and tree height, can be then
calculated. Similarly, [149] adopted a deep learning network for individual
tree detection from urban street images and depended on the single-view depth
estimation and triangulation techniques to calculate the precise tree geo-
localization.

The underwater image collection is a frequent task for marine exploration,
which has been completed by using autonomous underwater vehicles. Acoustic
sensors are widely utilized instruments for obstacle avoidance. However, they
cannot work well at a close range. In the work of [150], an obstacle detection
instrument has been designed by using a spherical camera, which can generate
real-time point clouds from a visual SLAM system and assist the autonomous
obstacle detection, as presented in Fig. 23(b).

Figure 23: Spherical cameras for (a) urban tree detection[148] and (b)
obstacle avoidance of underwater vehicles[150].

##  IV Prospects

3D reconstruction from spherical images has been greatly promoted by the
development of professional and consumer-grade cameras and automatic image
processing techniques, as well as their increasing usage in varying fields.
Compared with classical perspective images, there still exist some challenges
for 3D reconstruction based on spherical images, which can be categorized as
exploitation of crowdsource image, spatial resolution of spherical image,
camera distortion and calibration, and integration of aerial-ground image. The
details are presented in the following subsections.

###  IV-A Exploitation of crowdsource image

In recent years, spherical images can be easily recorded based on low-cost and
easy-to-use spherical cameras, as presented in Fig. 4. Besides, spherical
images can also be downloaded from well-known map providers, e.g., Google Maps
and Tencent Maps. These crowdsource images are usually freely provided for
non-commercial usage, which plays a critical role in spherical image-based
applications[12]. However, 3D reconstruction based on these crowdsource images
is non-trivial for two main reasons. On the one hand, the diversity of
crowdsource images is very large, which can be recorded by using professional
or consumer-grade cameras, and obtained from indoor or outdoor scenes, as
shown in Fig. 24; on the other hand, the overlap degree of crowdsource images
can not be ensured since they are captured by non-professional users or just
for the visual navigation purpose. These factors can frequently cause the
failure of 3D reconstruction.

Crowdsource images still have great potential in 3D reconstruction, especially
for urban buildings, although they have the above-mentioned drawbacks. There
are some possible solutions to exploit the potential of freely available
crowdsource images. First, image classification methods can be used to
separate indoor and outdoor images, e.g., recent deep learning-based networks.
The classified images can be used for subsequent indoor or outdoor 3D
modeling. Second, existing crowdsource images can be useful compensation from
the street viewpoints, such as for UAV images from aerial viewpoints. In this
situation, crowdsource images could be registered to the reconstructed 3D
models from UAV images, avoiding the requirement of high overlap degrees in
image-based 3D reconstruction.

Figure 24: Crowdsource images from Google Street Views.

###  IV-B Spatial resolution of spherical image

Spherical cameras record the 360 by 180 degrees of surrounding environments
using only one shot, which dramatically accelerates data acquisition. However,
because of the limited CCD (Charge Coupled Device) dimensions of cameras, the
spatial resolution of collected images is also obviously decreased when
compared with classical perspective cameras with images of the same CCD
dimensions. The spatial resolution causes a contradiction between the
efficiency of data acquisition and the quality of the reconstructed model.

  * •

The efficiency of data acquisitions. Much more time would be consumed to
record images with higher resolutions to ensure the enough camera exposure
time. For example, a Weiss AG Civetta spherical camera can record images with
a dimension of 230 M pixels as it uses a rotation lens to scan surrounding
scenes. This camera, however, can only be installed on a ground-fixed tripod
since it consumes 40 seconds for each camera exposure.

  * •

The quality of reconstructed models. The consumer-grade cameras usually
consist of two or more dioptric camera modules, such as the Theta X camera
with two lenses. This design model decreases the time costs consumed in
outdoor data acquisition. However, they sacrifice the spatial resolution of
recorded images, which can further reduce the quality of reconstructed models.

In order to increase the spatial resolution of spherical images, two possible
solutions can be implemented without sacrificing the efficiency of data
acquisition. On the one hand, more cameras can be integrated into one
spherical camera as the spherical image is the fusion of images from these
camera modules. This strategy has been widely used in recent spherical
cameras, such as the Ladybug 5+ with six cameras and the Panono with 36
cameras; on the other hand, high-speed imaging could be a promising
technology. It does not increase the size and weight of spherical cameras, and
it decreases the time costs for recording high-resolution images[151].

###  IV-C Camera distortion and calibration

Serious geometric distortions exist in spherical images. On the one hand, the
individual camera module has imaging geometric distortions; on the other hand,
generating the spherical image would also introduce distortions. Especially
for consumer-grade cameras, such as Ricoh Theta X and Insta360 Sphere, these
cameras project multiple images from each camera onto a sphere and generate
the ERP format spherical images. Generally, an ideal unit sphere camera model
is used in the generation of the ERP format, as illustrated in Fig. 13(c),
including the well-known commercial and open-source software packages Agisoft
Metashape, Pix4Dmapper, and OpenMVG. This projection would introduce
distortions and cause errors in 3D reconstruction. Fig. 25 presents a
comparison of 3D models reconstructed from consumer-grade and professional
spherical images. It is clearly shown that the point clouds in Fig. 25(a) are
very coarse, even on the plain wall. On the contrary, the point clouds
generated using the professional camera are more accurate, from which the text
is very clear. Therefore, camera calibration should be seriously considered
and executed for spherical images to improve the quality of reconstructed
models.

Figure 25: The illustration of dense matching point clouds from (a) a
consumer-grade camera; and (b) a professional camera.

###  IV-D Integration of aerial-ground image

Although spherical cameras can record 360 by 180 degrees of surrounding
environments, their effective observation range is still limited to the near-
ground region. In order to obtain complete reconstruction, aerial images must
be integrated with ground spherical images. In the literature, there are
valuable attempts for the integration of aerial-ground images in the context
of 3D reconstruction, which can be roughly divided into 2D image-based methods
and 3D point cloud-based methods[152].

  * •

2D image-based methods. These methods aim to establish reliable correspondence
between aerial and ground images and implement combined bundle adjustment by
using both aerial and ground images. The commonly used strategies are image
rectification and virtual rendering. The former aims to rectify aerial and
ground images onto the same plane in the object space[153]; the latter
utilizes the rough POS (Positioning and Orientation System) data of ground
images and render virtual images from the 3D models reconstructed from aerial
images[154]. The core idea of these methods is to decrease the geometric
distortion caused by large viewpoints. However, for spherical images, their
camera imaging model differs from the conventional perspective model. Thus,
further consideration should be paid.

  * •

3D point cloud-based methods. The core idea is to create 3D models separately
from aerial and ground images and find reliable enough common 3D
correspondences from these 3D models. The integration problem is then
converted into the registration of 3D point clouds[152]. However, for
spherical images, two issues should be considered. On the one hand, the
trajectory of spherical images is limited by the structure of urban streets,
which causes large accumulate drift in SfM or SLAM-based image orientation.
Thus, it may be hard to model the registration between 3D point clouds by
using the commonly used similarity transformation; on the other hand, as shown
in Fig. 25, the reconstructed models of spherical images may contain many
false 3D points due to camera distortions. Finding reliable correspondences
becomes a non-trivial task between 3D point clouds created from varying
images.

For the first issue, the recent deep learning-based technique can be used to
achieve reliable feature detection and matching. Especially for decreasing the
projection distortions, the DCN (deformable convolutional network)[155] based
network can be a promising technique to achieve viewpoint invariant feature
detection, and the GCN (graph convolutional network) based network, e.g.,
SuperGlue[156], can exploit the context information to execute reliable
feature matching. For the second issue, the large image orientation problem
can be divided into some clusters based on the street structures, and the
accumulated drift can be decreased. This divide-and-conquer strategy has been
used for efficient and accurate 3D reconstruction of UAV images[157].

##  V Conclusions

Spherical images can record all surrounding environments by using one camera
exposure. In contrast to perspective images with limited FOV, spherical images
can cover the whole scene and have been increasingly used for 3D modeling in
street-view and indoor environments. This paper reviews the 3D reconstruction
from spherical images in terms of data acquisition, image matching, image
orientation, and dense matching to wrap up recent techniques of 3D modeling
based on spherical images. It also presents promising 3D reconstruction
applications, including cultural heritage documentation, urban modeling and
navigation, tunnel mapping and inspection, and other applications, e.g., urban
tree localization, emergency response and rescue, and underwater collision
avoidance. Finally, the main prospects are discussed in terms of the
exploitation of crowdsource images, the spatial resolution of spherical
images, camera distortion and calibration, and integration of aerial-ground
images. According to this review, we can conclude that spherical images have
great potential in the 3D reconstruction of street view and indoor scenes, and
contemporary techniques can support their applications. Future studies can
increase the spatial resolution of spherical cameras for data acquisition and
exploit current deep learning-based methods to optimize the 3D reconstruction
workflow.

##  VI Acknowledgment

The authors would like to thank the anonymous reviewers and editors, whose
comments and advice improved the quality of the work. This research was funded
by the National Natural Science Foundation of China (Grant No. 42001413) and
the Hong Kong Scholars Program (Grant No. 2021-114).

## References

  * [1] B. Xiong, M. Jancosek, S. O. Elberink, and G. Vosselman, “Flexible building primitives for 3d building modeling,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 101, pp. 275–290, 2015.
  * [2] A. Murtiyoso and P. Grussenmeyer, “Documentation of heritage buildings using close-range uav images: dense matching issues, comparison and case studies,” _The Photogrammetric Record_ , vol. 32, no. 159, pp. 206–229, 2017.
  * [3] J. Liao, Y. Yue, D. Zhang, W. Tu, R. Cao, Q. Zou, and Q. Li, “Automatic tunnel crack inspection using an efficient mobile imaging module and a lightweight cnn,” _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * [4] C. Zhang, Y. Cui, Z. Zhu, S. Jiang, and W. Jiang, “Building height extraction from gf-7 satellite images based on roof contour constrained stereo matching,” _Remote Sensing_ , vol. 14, no. 7, p. 1566, 2022.
  * [5] T. L. da Silveira and C. R. Jung, “Dense 3d scene reconstruction from multiple spherical images for 3-dof+ vr applications,” in _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2019, pp. 9–18.
  * [6] S. Ji, Z. Qin, J. Shan, and M. Lu, “Panoramic slam from a multiple fisheye camera rig,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 159, pp. 169–183, 2020.
  * [7] Z. Zhang, H. Rebecq, C. Forster, and D. Scaramuzza, “Benefit of large field-of-view cameras for visual odometry,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 801–808.
  * [8] N. Bruno and R. Roncella, “Accuracy assessment of 3d models generated from google street view imagery,” _The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 42, pp. 181–188, 2019.
  * [9] G. Fangi, R. Pierdicca, M. Sturari, and E. Malinverni, “Improving spherical photogrammetry using 360° omni-cameras: Use cases and new applications.” _International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences_, vol. 42, no. 2, 2018.
  * [10] S. Gao, K. Yang, H. Shi, K. Wang, and J. Bai, “Review on panoramic imaging and its applications in scene understanding,” _arXiv preprint arXiv:2205.05570_ , 2022.
  * [11] J.-P. Jhan, N. Kerle, and J.-Y. Rau, “Integrating uav and ground panoramic images for point cloud analysis of damaged building,” _IEEE geoscience and remote sensing letters_ , vol. 19, pp. 1–5, 2021.
  * [12] F. Biljecki and K. Ito, “Street view imagery in urban analytics and gis: A review,” _Landscape and Urban Planning_ , vol. 215, p. 104217, 2021.
  * [13] L. Cheng, Y. Yuan, N. Xia, S. Chen, Y. Chen, K. Yang, L. Ma, and M. Li, “Crowd-sourced pictures geo-localization method based on street view images and 3d reconstruction,” _ISPRS journal of photogrammetry and remote sensing_ , vol. 141, pp. 72–85, 2018.
  * [14] W. Wen, Y. Zhou, G. Zhang, S. Fahandezh-Saadi, X. Bai, W. Zhan, M. Tomizuka, and L.-T. Hsu, “Urbanloco: A full sensor suite dataset for mapping and localization in urban scenes,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 2310–2316.
  * [15] G. Fangi and C. Nardinocchi, “Photogrammetric processing of spherical panoramas,” _The photogrammetric record_ , vol. 28, no. 143, pp. 293–311, 2013.
  * [16] Z. Kang, J. Yang, Z. Yang, and S. Cheng, “A review of techniques for 3d reconstruction of indoor environments,” _ISPRS International Journal of Geo-Information_ , vol. 9, no. 5, p. 330, 2020.
  * [17] A. Pagani and D. Stricker, “Structure from motion using full spherical panoramic cameras,” in _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_. IEEE, 2011, pp. 375–382.
  * [18] T. Luhmann, “A historical review on panorama photogrammetry,” _International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 34, no. 5/W16, p. 8, 2004.
  * [19] D. Scaramuzza and K. Ikeuchi, “Omnidirectional camera,” 2014.
  * [20] T. L. da Silveira, P. G. Pinto, J. Murrugarra-Llerena, and C. R. Jung, “3d scene geometry estimation from 360 imagery: A survey,” _ACM Computing Surveys_ , vol. 55, no. 4, pp. 1–39, 2022.
  * [21] P. Moulon, P. Monasse, R. Perrot, and R. Marlet, “Openmvg: Open multiple view geometry,” in _International Workshop on Reproducible Research in Pattern Recognition_. Springer, 2016, pp. 60–74.
  * [22] Agisoft, “Agisoft metashape homepage,” http://www.agisoft.com, 2022, accessed: 2022-1-15.
  * [23] Pix4dMapper, “Pix4dmapper homepage,” https://www.pix4d.com, 2022, accessed: 2022-1-15.
  * [24] D. Anguelov, C. Dulong, D. Filip, C. Frueh, S. Lafon, R. Lyon, A. Ogale, L. Vincent, and J. Weaver, “Google street view: Capturing the world at street level,” _Computer_ , vol. 43, no. 6, pp. 32–38, 2010.
  * [25] S. Herban, D. Costantino, V. S. Alfio, and M. Pepe, “Use of low-cost spherical cameras for the digitisation of cultural heritage structures into 3d point clouds,” _Journal of Imaging_ , vol. 8, no. 1, p. 13, 2022.
  * [26] X. Zhang, P. Zhao, Q. Hu, M. Ai, D. Hu, and J. Li, “A uav-based panoramic oblique photogrammetry (pop) approach using spherical projection,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 159, pp. 198–219, 2020.
  * [27] W. AG, “Weiss ag homepage,” https://weiss-ag.com/, 2022, accessed: 2022-1-15.
  * [28] N. Zioulis, A. Karakottas, D. Zarpalas, and P. Daras, “Omnidepth: Dense depth estimation for indoors spherical panoramas,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 448–465.
  * [29] N.-H. Wang, B. Solarte, Y.-H. Tsai, W.-C. Chiu, and M. Sun, “360sd-net: 360 stereo depth estimation with learnable cost volume,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 582–588.
  * [30] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” _arXiv preprint arXiv:1709.06158_ , 2017.
  * [31] P. K. Lai, S. Xie, J. Lang, and R. Laganière, “Real-time panoramic depth maps from omni-directional stereo images for 6 dof videos in virtual reality,” in _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2019, pp. 405–412.
  * [32] F.-E. Wang, H.-N. Hu, H.-T. Cheng, J.-T. Lin, S.-T. Yang, M.-L. Shih, H.-K. Chu, and M. Sun, “Self-supervised learning of depth and camera motion from 360°videos,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 53–68.
  * [33] I. Armeni, S. Sax, A. R. Zamir, and S. Savarese, “Joint 2d-3d-semantic data for indoor scene understanding,” _arXiv preprint arXiv:1702.01105_ , 2017\.
  * [34] W. Hartmann, M. Havlena, and K. Schindler, “Recent developments in large-scale tie-point matching,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 115, pp. 47–62, 2016.
  * [35] S. Jiang, W. Jiang, B. Guo, L. Li, and L. Wang, “Learned local features for structure from motion of uav images: A comparative evaluation,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 14, pp. 10 583–10 597, 2021.
  * [36] J. Ma, X. Jiang, A. Fan, J. Jiang, and J. Yan, “Image matching from handcrafted to deep features: A survey,” _International Journal of Computer Vision_ , vol. 129, no. 1, pp. 23–79, 2021.
  * [37] Q. Zhao, W. Feng, L. Wan, and J. Zhang, “Sphorb: A fast and robust binary feature on the sphere,” _International journal of computer vision_ , vol. 113, no. 2, pp. 143–159, 2015.
  * [38] J. Cruz-Mota, I. Bogdanova, B. Paquier, M. Bierlaire, and J.-P. Thiran, “Scale invariant feature transform on the sphere: Theory and applications,” _International journal of computer vision_ , vol. 98, no. 2, pp. 217–241, 2012.
  * [39] M. Eder, M. Shvets, J. Lim, and J.-M. Frahm, “Tangent images for mitigating spherical distortion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 12 426–12 434.
  * [40] C. Jiang, J. Huang, K. Kashinath, P. Marcus, M. Niessner _et al._ , “Spherical cnns on unstructured grids,” _arXiv preprint arXiv:1901.02039_ , 2019.
  * [41] M. Defferrard, M. Milani, F. Gusset, and N. Perraudin, “Deepsphere: a graph-based spherical cnn,” _arXiv preprint arXiv:2012.15000_ , 2020.
  * [42] T. S. Cohen, M. Geiger, J. Köhler, and M. Welling, “Spherical cnns,” _arXiv preprint arXiv:1801.10130_ , 2018.
  * [43] B. Coors, A. P. Condurache, and A. Geiger, “Spherenet: Learning spherical representations for detection and classification in omnidirectional images,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 518–533.
  * [44] A. Torii, M. Havlena, and T. Pajdla, “From google street view to 3d city models,” in _2009 IEEE 12th international conference on computer vision workshops, ICCV Workshops_. IEEE, 2009, pp. 2188–2195.
  * [45] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” _Computer vision and image understanding_ , vol. 110, no. 3, pp. 346–359, 2008.
  * [46] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [47] J.-M. Morel and G. Yu, “Asift: A new framework for fully affine invariant image comparison,” _SIAM journal on imaging sciences_ , vol. 2, no. 2, pp. 438–469, 2009.
  * [48] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “Orb: An efficient alternative to sift or surf,” in _Computer Vision (ICCV), 2011 IEEE International Conference on_. IEEE, 2011, pp. 2564–2571.
  * [49] S. Jiang and W. Jiang, “On-board gnss/imu assisted feature extraction and matching for oblique uav images,” _Remote Sensing_ , vol. 9, no. 8, p. 813, 2017.
  * [50] Y. Wang, S. Cai, S.-J. Li, Y. Liu, Y. Guo, T. Li, and M.-M. Cheng, “Cubemapslam: A piecewise-pinhole monocular fisheye slam system,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 34–49.
  * [51] H. Taira, Y. Inoue, A. Torii, and M. Okutomi, “Robust feature matching for distorted projection by spherical cameras,” _IPSJ Transactions on Computer Vision and Applications_ , vol. 7, pp. 84–88, 2015.
  * [52] T. Chuang and N. Perng, “Rectified feature matching for spherical panoramic images,” _Photogrammetric Engineering & Remote Sensing_, vol. 84, no. 1, pp. 25–32, 2018.
  * [53] H. Guan and W. A. Smith, “Brisks: Binary features for spherical images on a geodesic grid,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4516–4524.
  * [54] Z. Arican and P. Frossard, “Scale-invariant features and polar descriptors in omnidirectional imaging,” _IEEE Transactions on Image Processing_ , vol. 21, no. 5, pp. 2412–2423, 2012.
  * [55] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg, “Matchnet: Unifying feature and metric learning for patch-based matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3279–3286.
  * [56] V. Kumar BG, G. Carneiro, and I. Reid, “Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5385–5394.
  * [57] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and F. Moreno-Noguer, “Discriminative learning of deep convolutional feature point descriptors,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 118–126.
  * [58] Z. Luo, T. Shen, L. Zhou, J. Zhang, Y. Yao, S. Li, T. Fang, and L. Quan, “Contextdesc: Local descriptor augmentation with cross-modality context,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2527–2536.
  * [59] Z. Luo, T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao, T. Fang, and L. Quan, “Geodesc: Learning local descriptors by integrating geometry constraints,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 168–183.
  * [60] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas, “Working hard to know your neighbor’s margins: Local descriptor learning loss,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 4826–4837.
  * [61] Y. Tian, B. Fan, and F. Wu, “L2-net: Deep learning of discriminative patch descriptor in euclidean space,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 661–669.
  * [62] D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superpoint: Self-supervised interest point detection and description,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 224–236.
  * [63] M. Dusmanu, I. Rocco, T. Pajdla, M. Pollefeys, J. Sivic, A. Torii, and T. Sattler, “D2-net: A trainable cnn for joint detection and description of local features,” in _CVPR 2019 - IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [64] T. L. T. Da Silveira and C. R. Jung, “Evaluation of keypoint extraction and matching for pose estimation using pairs of spherical images,” in _2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_. IEEE, 2017, pp. 374–381.
  * [65] J. Murrugarra-Llerena, T. L. da Silveira, and C. R. Jung, “Pose estimation for two-view panoramas based on keypoint matching: A comparative study and critical analysis,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5202–5211.
  * [66] Y.-C. Su and K. Grauman, “Learning spherical convolution for fast features from 360 imagery,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [67] Q. Zhao, C. Zhu, F. Dai, Y. Ma, G. Jin, and Y. Zhang, “Distortion-aware cnns for spherical images.” in _IJCAI_ , 2018, pp. 1198–1204.
  * [68] Y. Shan and S. Li, “Descriptor matching for a discrete spherical image with a convolutional neural network,” _IEEE Access_ , vol. 6, pp. 20 748–20 755, 2018.
  * [69] Y.-C. Su and K. Grauman, “Kernel transformer networks for compact spherical convolution,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9442–9451.
  * [70] M. Muja and D. G. Lowe, “Fast approximate nearest neighbors with automatic algorithm configuration.” _VISAPP (1)_ , vol. 2, no. 331-340, p. 2, 2009\.
  * [71] S. Jiang, W. Jiang, and L. Wang, “Unmanned aerial vehicle-based photogrammetric 3d mapping: A survey of techniques, applications, and challenges,” _IEEE Geoscience and Remote Sensing Magazine_ , vol. 10, no. 2, pp. 135–171, 2022.
  * [72] S. Jiang, C. Jiang, and W. Jiang, “Efficient structure from motion for large-scale uav images: A review and a comparison of sfm tools,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 167, pp. 230–251, 2020.
  * [73] S. Jiang and W. Jiang, “Efficient match pair selection for oblique uav images based on adaptive vocabulary tree,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 161, pp. 61–75, 2020.
  * [74] L. Zheng, Y. Yang, and Q. Tian, “Sift meets cnn: A decade survey of instance retrieval,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 5, pp. 1224–1244, 2017.
  * [75] N. Snavely, S. M. Seitz, and R. Szeliski, “Photo tourism: exploring photo collections in 3d,” in _ACM transactions on graphics (TOG)_ , vol. 25. ACM, 2006, pp. 835–846.
  * [76] E. Mouragnon, M. Lhuillier, M. Dhome, F. Dekeyser, and P. Sayd, “Real time localization and 3d reconstruction,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 1. IEEE, 2006, pp. 363–370.
  * [77] C. Mei and P. Rives, “Single view point omnidirectional camera calibration from planar grids,” in _Proceedings 2007 IEEE International Conference on Robotics and Automation_. IEEE, 2007, pp. 3945–3950.
  * [78] L. Heng, B. Li, and M. Pollefeys, “Camodocal: Automatic intrinsic and extrinsic calibration of a rig with multiple generic cameras and odometry,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 1793–1800.
  * [79] D. Scaramuzza, A. Martinelli, and R. Siegwart, “A flexible technique for accurate omnidirectional camera calibration and structure from motion,” in _Fourth IEEE International Conference on Computer Vision Systems (ICVS’06)_. IEEE, 2006, pp. 45–45.
  * [80] S. Ji, Y. Shi, Z. Shi, A. Bao, J. Li, X. Yuan, Y. Duan, and R. Shibasaki, “Comparison of two panoramic sensor models for precise 3d measurements,” _Photogrammetric Engineering & Remote Sensing_, vol. 80, no. 3, pp. 229–238, 2014.
  * [81] S. Urban, J. Leitloff, and S. Hinz, “Improved wide-angle, fisheye and omnidirectional camera calibration,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 108, pp. 72–79, 2015.
  * [82] S. Aghayari, M. Saadatseresht, M. Omidalizarandi, and I. Neumann, “Geometric calibration of full spherical panoramic ricoh-theta camera,” _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences IV-1/W1 (2017)_ , vol. 4, pp. 237–245, 2017.
  * [83] D. D. Lichti, D. Jarron, W. Tredoux, M. Shahbazi, and R. Radovanovic, “Geometric modelling and calibration of a spherical camera imaging system,” _The Photogrammetric Record_ , vol. 35, no. 170, pp. 123–142, 2020.
  * [84] L. Puig, J. Bermúdez, P. Sturm, and J. J. Guerrero, “Calibration of omnidirectional cameras in practice: A comparison of methods,” _Computer Vision and Image Understanding_ , vol. 116, no. 1, pp. 120–137, 2012.
  * [85] M. Schönbein, T. Strauß, and A. Geiger, “Calibrating and centering quasi-central catadioptric cameras,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 4443–4450.
  * [86] J. Rehder, J. Nikolic, T. Schneider, T. Hinzmann, and R. Siegwart, “Extending kalibr: Calibrating the extrinsics of multiple imus and of individual axes,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 4304–4311.
  * [87] H. Cui, X. Gao, S. Shen, and Z. Hu, “Hsfm: Hybrid structure-from-motion,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1212–1221.
  * [88] M. Cao, W. Jia, Z. Lv, Y. Li, W. Xie, L. Zheng, and X. Liu, “Fast and robust feature tracking for 3d reconstruction,” _Optics and Laser Technology_ , vol. 110, pp. 120–128, 2019.
  * [89] N. Snavely, S. M. Seitz, and R. Szeliski, “Modeling the world from internet photo collections,” _International Journal of Computer Vision_ , vol. 80, no. 2, pp. 189–210, 2008.
  * [90] B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon, “Bundle adjustment—a modern synthesis,” in _International workshop on vision algorithms_. Springer, 1999, pp. 298–372.
  * [91] P. Chang and M. Hebert, “Omni-directional structure from motion,” in _Proceedings IEEE Workshop on Omnidirectional Vision (Cat. No. PR00704)_. IEEE, 2000, pp. 127–133.
  * [92] A. Torii, A. Imiya, and N. Ohnishi, “Two-and three-view geometry for spherical cameras,” in _Proceedings of the sixth workshop on omnidirectional vision, camera networks and non-classical cameras_. Citeseer, 2005, pp. 81–88.
  * [93] F. Kangni and R. Laganiere, “Orientation and pose recovery from spherical panoramas,” in _2007 IEEE 11th International Conference on Computer Vision_. IEEE, 2007, Conference Proceedings, pp. 1–8.
  * [94] B. Micusik and T. Pajdla, “Structure from motion with wide circular field of view cameras,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 28, no. 7, pp. 1135–1149, 2006.
  * [95] H. Guan and W. A. Smith, “Structure-from-motion in spherical video using the von mises-fisher distribution,” _IEEE Transactions on Image Processing_ , vol. 26, no. 2, pp. 711–723, 2016.
  * [96] C. Griwodz, S. Gasparini, L. Calvet, P. Gurdjos, F. Castan, B. Maujean, G. De Lillo, and Y. Lanthony, “Alicevision meshroom: An open-source 3d reconstruction pipeline,” in _Proceedings of the 12th ACM Multimedia Systems Conference_ , 2021, pp. 241–247.
  * [97] J. L. Schonberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4104–4113.
  * [98] E. Rupnik, M. Daakir, and M. Pierrot Deseilligny, “Micmac–a free, open-source solution for photogrammetry,” _Open Geospatial Data, Software and Standards_ , vol. 2, no. 1, pp. 1–9, 2017.
  * [99] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [100] S. Sumikura, M. Shibuya, and K. Sakurada, “Openvslam: A versatile visual slam framework,” in _Proceedings of the 27th ACM International Conference on Multimedia_ , 2019, pp. 2292–2295.
  * [101] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 3, pp. 611–625, 2017.
  * [102] S. Urban and S. Hinz, “Multicol-slam-a modular real-time multi-camera slam system,” _arXiv preprint arXiv:1610.07336_ , 2016.
  * [103] J. Engel, T. Schöps, and D. Cremers, “Lsd-slam: Large-scale direct monocular slam,” in _European conference on computer vision_. Springer, 2014, pp. 834–849.
  * [104] B. Huang, J. Zhao, and J. Liu, “A survey of simultaneous localization and mapping,” _arXiv preprint arXiv:1909.05214_ , 2019.
  * [105] F. Huang, H. Yang, X. Tan, S. Peng, J. Tao, and S. Peng, “Fast reconstruction of 3d point cloud model using visual slam on embedded uav development platform,” _Remote Sensing_ , vol. 12, no. 20, p. 3308, 2020.
  * [106] Y. Lu, Z. Xue, G.-S. Xia, and L. Zhang, “A survey on vision-based uav navigation,” _Geo-spatial information science_ , vol. 21, no. 1, pp. 21–32, 2018.
  * [107] D. Caruso, J. Engel, and D. Cremers, “Large-scale direct slam for omnidirectional cameras,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 141–148.
  * [108] H. Matsuki, L. Von Stumberg, V. Usenko, J. Stückler, and D. Cremers, “Omnidirectional dso: Direct sparse odometry with fisheye cameras,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3693–3700, 2018\.
  * [109] C. Won, H. Seok, Z. Cui, M. Pollefeys, and J. Lim, “Omnislam: Omnidirectional localization and dense mapping for wide-baseline multi-camera systems,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 559–566.
  * [110] J.-P. Tardif, Y. Pavlidis, and K. Daniilidis, “Monocular visual odometry in urban environments using an omnidirectional camera,” in _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2008, pp. 2531–2538.
  * [111] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “Orb-slam: a versatile and accurate monocular slam system,” _IEEE Transactions on Robotics_ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [112] R. Mur-Artal and J. D. Tardos, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE Transactions on Robotics_ , pp. 1–8, 2017.
  * [113] H. Huang and S.-K. Yeung, “360vo: Visual odometry using a single 360 camera,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2022.
  * [114] S. Im, H. Ha, F. Rameau, H.-G. Jeon, G. Choe, and I. S. Kweon, “All-around depth from small motion with a spherical panoramic camera,” in _European Conference on Computer Vision_. Springer, 2016, pp. 156–172.
  * [115] Y. Zhang and F. Huang, “Panoramic visual slam technology for spherical images,” _Sensors_ , vol. 21, no. 3, p. 705, 2021.
  * [116] K. Chappellet, G. Caron, F. Kanehiro, K. Sakurada, and A. Kheddar, “Benchmarking cameras for open vslam indoors,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 4857–4864.
  * [117] Q. Feng, H. P. Shum, and S. Morishima, “360 depth estimation in the wild-the depth360 dataset and the segfuse network,” in _2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_. IEEE, 2022, pp. 664–673.
  * [118] H. Jang, A. Meuleman, D. Kang, D. Kim, C. Richardt, and M. H. Kim, “Egocentric scene reconstruction from an omnidirectional video,” _ACM Transactions on Graphics (TOG)_ , vol. 41, no. 4, pp. 1–12, 2022.
  * [119] L. Jin, Y. Xu, J. Zheng, J. Zhang, R. Tang, S. Xu, J. Yu, and S. Gao, “Geometric structure based and regularized depth estimation from 360 indoor imagery,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 889–898.
  * [120] G. Pintore, M. Agus, E. Almansa, J. Schneider, and E. Gobbetti, “Slicenet: deep dense depth estimation from a single indoor panorama using a slice-based representation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 536–11 545.
  * [121] A. Pagani, C. C. Gava, Y. Cui, B. Krolla, J.-M. Hengen, and D. Stricker, “Dense 3d point cloud generation from multiple high-resolution spherical images.” in _VAST_ , 2011, pp. 17–24.
  * [122] Y. Furukawa, C. Hernández _et al._ , “Multi-view stereo: A tutorial,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 9, no. 1-2, pp. 1–148, 2015.
  * [123] M. Bleyer, C. Rhemann, and C. Rother, “Patchmatch stereo-stereo matching with slanted support windows.” in _Bmvc_ , vol. 11, 2011, pp. 1–11.
  * [124] Y. Furukawa and J. Ponce, “Accurate, dense, and robust multiview stereopsis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 8, no. 32, pp. 1362–1376, 2010.
  * [125] H. Hirschmuller, “Stereo processing by semiglobal matching and mutual information,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 30, no. 2, pp. 328–341, 2007.
  * [126] M. Rothermel, K. Wenzel, D. Fritsch, and N. Haala, “Sure: Photogrammetric surface reconstruction from imagery,” in _Proceedings LC3D Workshop, Berlin_ , vol. 8, no. 2, 2012.
  * [127] J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5410–5418.
  * [128] S. Khamis, S. Fanello, C. Rhemann, A. Kowdle, J. Valentin, and S. Izadi, “Stereonet: Guided hierarchical refinement for real-time edge-aware depth prediction,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 573–590.
  * [129] A. Seki and M. Pollefeys, “Sgm-nets: Semi-global matching with neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 231–240.
  * [130] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for unstructured multi-view stereo,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 767–783.
  * [131] Y. Yao, Z. Luo, S. Li, J. Zhang, Y. Ren, L. Zhou, T. Fang, and L. Quan, “Blendedmvs: A large-scale dataset for generalized multi-view stereo networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1790–1799.
  * [132] H. Kim and A. Hilton, “3d scene reconstruction from multiple spherical stereo pairs,” _International journal of computer vision_ , vol. 104, no. 1, pp. 94–116, 2013.
  * [133] A. Meuleman, H. Jang, D. S. Jeon, and M. H. Kim, “Real-time sphere sweeping stereo from multiview fisheye images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 11 423–11 432.
  * [134] G. Fangi, “The multi-image spherical panoramas as a tool for architectural survey-xxi international cipa symposium, 1-6 october 2007, atene, isprs international archive–vol xxxvi-5/c53–issn 1682-1750–cipa archives vol,” _XXI-2007 ISSN_ , pp. 0256–1840, 2007.
  * [135] G. Fangi, “Multiscale multiresolution spherical photogrammetry with long focal lenses for architectural surveys,” _International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 38, no. Part 5, pp. 1–6, 2010.
  * [136] L. Barazzetti, M. Previtali, and F. Roncoroni, “Can we use low-cost 360 degree cameras to create accurate 3d models?” _International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences_, vol. 42, no. 2, 2018.
  * [137] B. Micusik and J. Kosecka, “Piecewise planar city 3d modeling from street view panoramic sequences,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2009, pp. 2906–2912.
  * [138] Z. Xu, F. Zhang, Y. Wu, Y. Yang, and Y. Wu, “Building height calculation for an urban area based on street view images and deep learning,” _Computer-Aided Civil and Infrastructure Engineering_ , 2022.
  * [139] M. Jayasuriya, R. Ranasinghe, and G. Dissanayake, “Active perception for outdoor localisation with an omnidirectional camera,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4567–4574.
  * [140] Q. Li, W. Yu, and S. Jiang, “Optimized views photogrammetry: Precision analysis and a large-scale case study in qingdao,” _arXiv preprint arXiv:2206.12216_ , 2022.
  * [141] M. Leingartner, J. Maurer, A. Ferrein, and G. Steinbauer, “Evaluation of sensors and mapping approaches for disasters in tunnels,” _Journal of field robotics_ , vol. 33, no. 8, pp. 1037–1057, 2016.
  * [142] Z.-H. Zhu, J.-Y. Fu, J.-S. Yang, and X.-M. Zhang, “Panoramic image stitching for arbitrarily shaped tunnel lining inspection,” _Computer-Aided Civil and Infrastructure Engineering_ , vol. 31, no. 12, pp. 936–953, 2016.
  * [143] M. Janiszewski, M. Torkan, L. Uotinen, and M. Rinne, “Rapid photogrammetry with a 360-degree camera for tunnel mapping,” _Remote Sensing_ , vol. 14, no. 21, p. 5494, 2022.
  * [144] X. Zhang, P. Zhao, Q. Hu, H. Wang, M. Ai, and J. Li, “A 3d reconstruction pipeline of urban drainage pipes based on multiviewimage matching using low-cost panoramic video cameras,” _Water_ , vol. 11, no. 10, p. 2101, 2019\.
  * [145] X. Fang, Q. Li, J. Zhu, Z. Chen, D. Zhang, K. Wu, K. Ding, and Q. Li, “Sewer defect instance segmentation, localization, and 3d reconstruction for sewer floating capsule robots,” _Automation in Construction_ , vol. 142, p. 104494, 2022.
  * [146] W. Guo, X. Zhang, D. Zhang, Z. Chen, B. Zhou, D. Huang, and Q. Li, “Detection and classification of pipe defects based on pipe-extended feature pyramid network,” _Automation in Construction_ , vol. 141, p. 104399, 2022.
  * [147] M. Karkoub, O. Bouhali, and A. Sheharyar, “Gas pipeline inspection using autonomous robots with omni-directional cameras,” _IEEE Sensors Journal_ , vol. 21, no. 14, pp. 15 544–15 553, 2020.
  * [148] K. Itakura and F. Hosoi, “Automatic tree detection from three-dimensional images reconstructed from 360 spherical camera using yolo v2,” _Remote Sensing_ , vol. 12, no. 6, p. 988, 2020.
  * [149] S. Lumnitz, T. Devisscher, J. R. Mayaud, V. Radic, N. C. Coops, and V. C. Griess, “Mapping trees along urban street networks with deep learning and street-level imagery,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 175, pp. 144–157, 2021.
  * [150] E. Ochoa, N. Gracias, K. Istenič, J. Bosch, P. Cieślak, and R. García, “Collision detection and avoidance for underwater vehicles using omnidirectional vision,” _Sensors_ , vol. 22, no. 14, p. 5354, 2022\.
  * [151] T. Feng, H. Mi, M. Scaioni, G. Qiao, P. Lu, W. Wang, X. Tong, and R. Li, “Measurement of surface changes in a scaled-down landslide model using high-speed stereo image sequences,” _Photogrammetric Engineering & Remote Sensing_, vol. 82, no. 7, pp. 547–557, 2016.
  * [152] X. Gao, L. Hu, H. Cui, S. Shen, and Z. Hu, “Accurate and efficient ground-to-aerial model alignment,” _Pattern Recognition_ , vol. 76, pp. 288–302, 2018.
  * [153] B. Wu, L. Xie, H. Hu, Q. Zhu, and E. Yau, “Integration of aerial oblique imagery and terrestrial imagery for optimized 3d modeling in urban areas,” _ISPRS journal of photogrammetry and remote sensing_ , vol. 139, pp. 119–132, 2018.
  * [154] Q. Zhu, Z. Wang, H. Hu, L. Xie, X. Ge, and Y. Zhang, “Leveraging photogrammetric mesh models for aerial-ground feature point matching toward integrated 3d reconstruction,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 166, pp. 26–40, 2020.
  * [155] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 9308–9316.
  * [156] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, “Superglue: Learning feature matching with graph neural networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 4938–4947.
  * [157] S. Jiang, Q. Li, W. Jiang, and W. Chen, “Parallel structure from motion for uav images via weighted connected dominating set,” _arXiv preprint arXiv:2206.11499_ , 2022.
  * [158] I. Puente, H. González-Jorge, J. Martínez-Sánchez, and P. Arias, “Review of mobile mapping and surveying technologies,” _Measurement_ , vol. 46, no. 7, pp. 2127–2145, 2013.
  * [159] J. Sun, Z. Shen, Y. Wang, H. Bao, and X. Zhou, “Loftr: Detector-free local feature matching with transformers,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 8922–8931.
  * [160] Y. Cui, Q. Chang, Q. Liu, X. Yang, Y. Huang, S. Chen, F. Ren, and D. Stricker, “3d reconstruction with spherical cameras,” _IEEE Access_ , 2021.
  * [161] S. Jiang and W. Jiang, “Efficient structure from motion for oblique uav images based on maximal spanning tree expansion,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 132, pp. 140–161, 2017.
  * [162] C. Wu, “Siftgpu: A gpu implementation of david lowe’s scale invariant feature transform (sift),” https://github.com/pitzer/SiftGPU, 2007, accessed: 2017-06-19.
  * [163] X. Gao, S. Shen, Z. Hu, and Z. Wang, “Ground and aerial meta-data integration for localization and reconstruction: A review,” _Pattern Recognition Letters_ , vol. 127, pp. 202–214, 2019.
  * [164] G. Li, X. Lu, B. Lin, L. Zhou, and G. Lv, “Automatic positioning of street objects based on self-adaptive constrained line of bearing from street-view images,” _ISPRS International Journal of Geo-Information_ , vol. 11, no. 4, p. 253, 2022.
  * [165] G. Yu and J.-M. Morel, “Asift: An algorithm for fully affine invariant comparison,” _Image Processing On Line_ , vol. 1, pp. 11–38, 2011.