# Estimating Extreme 3D Image Rotation with Transformer Cross-Attention

Shay Dekel, Yosi Keller
Bar Ilan University<EMAIL_ADDRESS>
###### Abstract

The estimation of large and extreme image rotation plays a key role in
multiple computer vision domains, where the rotated images are related by a
limited or a non-overlapping field of view. Contemporary approaches apply
convolutional neural networks to compute a 4D correlation volume to estimate
the relative rotation between image pairs. In this work, we propose a cross-
attention-based approach that utilizes CNN feature maps and a Transformer-
Encoder, to compute the cross-attention between the activation maps of the
image pairs, which is shown to be an improved equivalent of the 4D correlation
volume, used in previous works. In the suggested approach, higher attention
scores are associated with image regions that encode visual cues of rotation.
Our approach is end-to-end trainable and optimizes a simple regression loss.
It is experimentally shown to outperform contemporary state-of-the-art schemes
when applied to commonly used image rotation datasets and benchmarks, and
establishes a new state-of-the-art accuracy on these datasets. We make our
code publicly
available111https://anonymous.4open.science/r/AttExtremeRotation-A467/.

##  1 Introduction

Estimating the relative pose between a pair of images is a crucial task in
computer vision, which is used in various applications such as indoor
navigation, augmented reality, autonomous driving, 3D reconstruction [43, 39],
camera localization [5, 44, 48], simultaneous localization and mapping [12,
37], and novel view synthesis [34, 41]. The current approach to image
registration involves extracting features, matching them, and establishing
correspondences between them. However, this approach is ineffective for input
pairs with little or no overlap, making it difficult to establish sufficient
feature correspondences for matching, such as in the images shown in Fig. 1.

(a)

(b)

(c)

(d)

Figure 1: The estimation of extreme 3D image rotations. First row: Image pairs
with a small overlap. Second row: non-overlapping image pairs. The proposed
scheme estimates the relative rotation between the image pairs.

Many applications [47, 31, 1] require an accurate estimation of the rotations
between images. The common approach to estimating extreme 3D rotations for
images with very limited or no-overlap as in Fig. 1 relates to the seminal
work by Coughlan and Yuille [10] who introduced a technique that is premised
on linear structures within an image, arising primarily from three directions
that are mutually orthogonal, one vertical (building walls), and two
horizontal (ground-level pavements, roads, etc.). Similarly, the “Single View
Metrology” by Criminisi et al. [11] and its extensions [59, 25, 40], use
parallel lines in the image and their corresponding vanishing points [20] for
camera calibration. Moreover, the relative rotation of a camera can also be
estimated using illumination cues [2], by analyzing the directions of the
lighting and cast shadows.

In this work, we propose a novel deep learning-based method for determining
the relative rotation between two images that are related by significant and
extreme rotations. Our approach does not follow the classical formulations
[10, 11] that explicitly detect handcrafted image cues such as lines, shadows
and their corresponding vanishing points. Instead, our method involves
designing a deep neural network that performs direct regression of the
relative rotation from the input images.

Inspired by the recent successful applications of Transformers [51] in
computer vision tasks such as object detection [8] and image recognition [23],
we adapt a Transformer-Encoder to compute a stacked multihead attention to
encode the cross-attention between the two latent representations of the image
pairs. Thus, a Transformer-Encoder [51] computes an improved stacked
representation of the 4D correlation volume (4DCV) used in previous works [49,
38, 29, 22, 15, 16], where the 4DCVs were calculated by inner products.
Instead of using a single layer of $N^{2}$ inner products as in the 4DCV, the
proposed Transformer-Encoder-based approach leverages multi-head attention
with its advanced network architecture and additional parameters to better
encode the interactions between the entries of the activation maps.
Interestingly, the attention maps computed by our scheme, shown in Section
4.5, show that the Transformer-Encoder assigns high attention scores to image
regions containing rotation-informative image cues, emphasizing vertical and
horizontal lines. We also observe that the proposed approach can predict the
rotation of non-overlapping image pairs with state-of-the-art (SOTA) accuracy.
Our framework is end-to-end trainable and optimizes a regression loss. It is
evaluated on three data-set benchmarks: StreetLearn [35], SUN360 [52] and
InteriorNet [28], with different classes of overlap in indoor and outdoor
locations and under varying illumination. The experimental results in Section
4 show our model to provide state-of-the-art (SOTA) accuracy.

In summary, our contributions are as follows:

  * •

We address the problem of estimating rotations in extreme scenarios, even with
little or no overlap.

  * •

We propose a cross-attention approach based on a Transformer-Encoder to encode
the pairwise attentions between the two latent spaces of the image pairs.

  * •

We found that the relative rotation can be estimated based on the cues from a
single image.

  * •

Our approach has been experimentally shown to compare favorably with state-of-
the-art rotation estimation schemes when applied to indoor and outdoor
datasets.

##  2 Related Work

Our rotations estimation approach is a particular instance of relative pose
estimation in general, and relative pose regression (RPR), in particular. The
common relative rotation estimation approach is to detect and match 2D feature
points [33, 4] in both images. In pose localization tasks [48], PnP schemes
are used to estimate the relative 3D rotation, and the camera pose of the
query image is estimated given the 3D coordinates and pose of the anchor
image. Recent approaches applied end-to-end trainable deep network to both
images [3, 14]. Graph neural networks (GNNs) were applied to multi-image RPR
by aggregating the localization cues from multiple video frames [53, 50].
Recently, neural radiation fields (NeRFs) were also applied to encode scenes
for RPR, instead of storing images or feature points. Some schemes only
estimate relative 3D rotations using rotation-specific parametrizations such
as quaternions and Euler angles [57] by formulating the problem as a $L_{2}$
regression using neural networks [57, 26]. Quaternions are used to resolve the
inherent discontinuities in the rotation representation due to their Double-
Cover property. Levinson et al. [26] studied the use of SVD orthogonalization
to estimate 3D rotations using neural networks. The SVD is applied to project
the estimated rotation matrices onto the rotation group. Their analysis
demonstrates that simply replacing traditional representations with the SVD
orthogonalization method leads to state-of-the-art results in various deep-
learning applications. The probability of distributions of the set of 3D
rotations was estimated by Mohlin et al. [36] that proposed to use a neural
network to estimate the parameters of the Fisher distribution matrix. The
optimization of the negative log-likelihood loss for this distribution leads
to improved results, outperforming the previous state-of-the-art on several
practical datasets. Applying a negative log-likelihood loss for this
distribution allows for improved rotation estimation and was successfully used
for head pose estimation by Liu et al. [32]. Rockwell et al. [42] present a
baseline to directly estimate the relative pose between two images by training
a Vision Transformer (ViT) to bring its computations close to the eight-point
algorithm. They achieve competitive results in multiple settings.

The methods discussed above require a significant overlap in the pair of input
images, where large rotation (and consequently) small overlap may lead to
inaccurate estimates. In particular, non-overlapping images can’t be aligned
using such schemes. Caspi and Irani [9] showed that two image sequences with
non-overlapping fields of view can be aligned both in time and in space, when
the two cameras are closely attached, based on common temporal changes within
the two sequences. Similarly, Shakil [45] showed that two (or more)
nonoverlapping videos recorded by uncalibrated video cameras can be aligned
using the temporal changes and frame-to-frame motion within the sequences. The
registration of non-overlapping RGB-D scans [46, 21] is a common extension of
the image-only problem, where a global representation of a scene is often
estimated.

Our work follows the setup studied by Cai et al. [7] where the extreme
relative 3D rotation is estimated given only a pair of input images. Cai et
al. propose to apply a CNN to embed the input images and a 4D correlation
volume of the image embeddings is computed. The 3D rotation is estimated by
applying an MLP to the correlation volume and optimizing a cross-entropy loss.
They achieved SOTA accuracy among non-overlapping images. 4D correlation
volumes (4DCV) are 4D representations of Bilinear Pooling [30, 24] that encode
all of the pairwise correlation between all entries of the 2D embedding maps
of a pair of images. Therefore, 4DCVs have been applied in tasks that require
long-range (spatial) correspondences, such as the RAFT SOTA optical flow [49],
as well as other optical flow schemes [22, 16, 54]. Similarly, 3D correlation
volumes were applied to deep stereo matching [38, 29, 19, 27], where each
pixel in one of the images is matched to limited spatial support in the other
image. In contrast, we propose to compute the equivalent of the 4DCV by
computing the cross-attention between the activation maps of the pair of input
images using a stacked multi-head Transformer-Encoder and a corresponding
activation mask. In particular, the Transformer-Encoder computes the
equivalent of multiple stacked correlation volumes [51] using MHA.

Figure 2: An overview of our method. The of input images
$(I_{1},I_{2})\in\mathbb{R}^{H\times W}$ are encoded by a weight-sharing
Siamese CNN to extract the feature maps $(\hat{I_{1}},\hat{I_{2}})$. These
feature maps are concatenated as ${T}$, and fed into a Transformer-Encoder
alongside the mask $M$ to compute their cross-attention $\hat{T}$. The first
half of $\hat{T}$ is reshaped and analyzed by the two residual blocks in Table
1. Finally, a fully connected layer predicts the relative rotation encoded by
the Quaternion $q$.

##  3 Rotation Estimation Using Cross-Attention

The proposed scheme estimates the relative 3D rotation
$\mathbf{R}\in\mathbb{R}^{3\times 3}$ between a pair of input images
$\\{I_{1},I_{2}\\}$, and is outlined in Fig. 2. Each of the input images is
encoded into the activation maps
$\\{\widehat{I}_{1},\widehat{I}_{2}\\}$$\in\mathbb{R}^{c\times K_{1}\times
K_{2}}$ using the weight-sharing Siamese Residual-Unet (ResUNet) [55] network,
where $c$ is the number of channels of the activation maps, while $K_{1}$ and
$K_{2}$ are their spatial dimensions. We compute an improved equivalent of the
4D correlation volume used in previous works [49, 7] between the input images
$I_{1}$ and $I_{2},$ by computing the cross-attention using a Transformer-
Encoder and a corresponding attention mask $\mathbf{M}$, as detailed in
Section 3.1. For that, we first row-vectorize $\widehat{I}_{1}$ and
$\hat{I_{2}}$ into two sequences $\in\mathbb{R}^{c\times K_{1}K_{2}}$ by
flattening the leading spatial dimension of each activation map. The sequences
are concatenated to a single tensor $T$$\in\mathbb{R}^{c\times 2K_{1}K_{2}}$
that is input to the Transformer-Encoder. The output sequence
$\widehat{\mathbf{T}}$ of the Transformer-Encoder is further analyzed using an
MLP, and the 3D rotation is regressed using its quaternion representation
$\mathbf{q,}$ as discussed in Section 3.2.

###  3.1 Cross-Attention computation using a Transformer-Encoder

The cross-attention between the pair of input images $I_{1}$ and $I_{2}$ is
computed using a Transformer-Encoder with $l=2$ layers and $h=4$ attention
heads for each layer. An ablation study of this configuration is given in
Section 4.7. The cross-attention maps computed by the Transformer-Encoder are
an improved equivalent of the 4D correlation volumes [49, 7], encoding the
interactions (inner-products) between all the image cues in the activation
maps. By default, a Transformer-Encoder computes the self-attention maps of
the input sequence. Hence, the cross-attention of the vectorized and
concatenated activation maps $T$ is computed by applying the attention mask
$\mathbf{M}$ given in Eq. 1. The mask $\mathbf{M}$ nullifies the self-
attention terms in the attention maps computed throughout the Transformer-
Encoder, while retaining the cross-attention terms,

$\mathbf{M}=\begin{bmatrix}-\infty&-\infty&\cdots&0&0\\\
-\infty&-\infty&\cdots&0&0\\\ \vdots&\vdots&\ddots&\vdots&\\\
0&0&\cdots&-\infty&-\infty\\\ 0&0&\cdots&-\infty&-\infty\end{bmatrix}$ (1)

The use of the mask $\mathbf{M}$ and the corresponding structure of the
attention maps is shown in Fig. 3.

Figure 3: Computing the cross-attention using a Transformer-Encoder and the
input mask, $\mathbf{M.}$ The mask $\mathbf{M}$ zeros the self-attention
terms, retaining only the cross-attention terms.

Any pair of image patches could hold valuable information about the overall
geometric relationships in an image. The Transformer-Encoder is able to
uncover these hints implicitly. The regions in the input images that contain
rotation-related cues, explicitly or implicitly, receive higher attention
scores, as seen in Section and Fig. 5. This leads to a more meaningful and
concise input for the subsequent MLP layer, ultimately resulting in improved
estimation accuracy. The same as in [7], even when the image pairs are non-
overlapping, the Transformer-Encoder formulation can predict the rotation
using straight lines only present in a single image, the same as human
cognitive capabilities. For example, in the extreme scenario of non-
overlapping image pairs the roll angle can be estimated from a single image,
by implicitly assuming that buildings and their edges are perpendicular to
ground level. Similarly, the relative elevation angle can be estimated by
assuming that the streets and pavements are parallel to the ground plane or
computing the corresponding vanishing points. Most training and test datasets
in this domain depict urban scenes, adhering to these assumptions.

###  3.2 Relative Rotation Regression

The refined representations $\hat{T}$ are used as input to the MLP regressor
outlined in Table 1 to determine the output rotation as the quaternion
$\mathbf{q}=[qw,qx,qy,qz]$. The training loss is calculated as follows:

$\mathbf{L}={||q_{0}-\frac{q}{||q||}||_{2}}.$ (2)

$q_{0}$ and $q$ are the groundtruth and predicted quaternions, respectively.
They are normalized to ensure that the quaternion is a valid representation of
a 3D rotation.

Layers | Residual1 |  Residual2
---|---|---
| $(k,s,p,N)$ | $(k,s,p,N)$
Batch norm |  |
Conv | $(1,1,0,128)$ | $(1,1,0,64)$
Batch norm |  |
Conv | $(3,2,1,128)$ | $(3,2,1,64)$
Batch norm |  |
Conv | $(1,1,0,512)$ | $(1,1,0,256)$
Average pooling | $(2,2)$ | $(2,2)$
Table 1: The architecture of the residual blocks in Fig. 2. For the
convolution layers $(k,s,p,N)$ relate to the filter supports $(k,k)$, strides
$(s,s)$, padding $(p,p)$ and the number of filters $N$. For the average
pooling layer, we report the pooling support $(k,k)$.

##  4 Experimental Results

The proposed scheme was experimentally verified by applying it to contemporary
benchmark datasets with overlapping and non-overlapping image pairs. We
followed the comprehensive experimental setup proposed by Cai et al. [7],
using the same datasets and image overlap classes as their study. In
particular, we used their source
code333https://github.com/RuojinCai/ExtremeRotation_code to create perspective
views from panoramic images, ensuring that the input images were the same in
both studies, allowing fair comparisons with previous SOTA results and other
contemporary schemes. For that, we also used the same Residual-Unet backbone
network [56] as in [7]. Section 4.1 details the image datasets we used and
their processing, in accordance with Cai et al. [7], to derive the training
and test datasets. Training details are given in Section 4.2. We compare with
the recent SOTA schemes listed in Section 4.3 using the geodesic error measure
used in previous works [7]

$\mathbf{E}=\arccos\left(\frac{tr(\mathbf{R}^{T}\mathbf{R}^{\ast})-1}{2}\right),$
(3)

where $\mathbf{R}$ is the predicted rotation matrix and $\mathbf{R}^{\ast}$ is
the groundtruth relative rotation matrix for each image pair. The experimental
comparisons are reported in Section 4.4 and the attention maps are visualized
in Section 4.5 to provide an intuitive interpretation of the cross-attention
scores computed by the Transformer-Encoder. We studied the cross-dataset
generalization properties of the proposed scheme in Section 4.6, while
ablation studies of the different parameters, design choices, and parameters
are reported in Section 4.7.

###  4.1 Image Datasets and their Processing

We used the following datasets and train/test splits used in previous works:

InteriorNet [28] is a synthetic dataset for understanding and mapping interior
scenes. A subset of 10,050 panoramas from 112 different houses was used, with
82 images used for training and 32 for testing, respectively.

StreetLearn [35] is an outdoor dataset consisting of close to 140,000
panoramic views of the cities of Pittsburgh and Manhattan. The dataset was
divided into 50K panoramic views for training and 3K panoramic views for
testing.

SUN360 [52] is an indoor collection of high-resolution panoramas that cover a
full view of $360^{\circ}\times 180^{\circ}$ for a variety of environmental
scenes downloaded from the Internet. It also provides location category
labels. We used 7K and 2K panoramas for training and testing, respectively.

As these datasets contain panoramic images, we generated 200 perspective
$128\times 128$ images by randomly cropping 200 different locations in each
panoramic image. Using such sampling allows us to generate uniformly
distributed ground-truth image pairs within a pitch resolution of
$[0^{\circ},90^{\circ}]$ and a Yaw resolution of $[-180^{\circ},180^{\circ}]$.
We estimate only the Yaw and the Pitch angles, assuming zero roll between the
image pairs. We avoided generating textureless image pairs, that is, images
that mainly contain ceilings or floors in a house or skies in outdoor scenes,
by limiting the pitch rang to $[-45^{\circ},45^{\circ}]$ for the outdoor
dataset and $[-30^{\circ},30^{\circ}]$ for the indoor datasets. There is no
overlap between the train and test datasets. To assess our outcomes in
relation to prior research and to analyze the influence of camera translation
on our rotation estimation approach, we partitioned the InteriorNet and
StreetLearn datasets into two groups: images with and without camera
translations. The non-translated images were acquired by randomly selecting
pairs of cropped images from a single panorama. In contrast, the datasets that
include translations (known as StreetLearn-T and InteriorNet-T) were generated
by randomly selecting pairs of cropped images from different panoramas, where
translations are less than $3m$. However, our method did not estimate these
translations. We evaluated our performance in overlapping and nonoverlapping
pairs and use the setup of Cai et al. [7] by dividing the datasets into three
overlap classes:

Large, contains highly overlapping pairs up to relative rotations of
$45^{\circ}$

Small, contains pairs that partially overlap with relative rotation angles
$\in[45^{\circ},90^{\circ}]$

None, contains pairs without overlap with relative rotations $>90^{\circ}$.

###  4.2 Training Details

We use a pre-trained Residual-Unet [56] (same as in Cai et al. [7]) as a
backbone to compute the feature maps of the two input images
$(\hat{I_{1}},\hat{I_{2}})$$\in\mathbb{R}^{c\times k_{1}\times
k_{2}}=\mathbb{R}$128×32×32. In line with Fig. 2, the embeddings were reshaped
and concatenated along the axis of the samples to form the tensor
$\mathbf{T}\in\mathbb{R}^{(2\cdot 32\cdot 32)\times
128}=\mathbb{R}^{2048\times 128}$. $\mathbf{T}$ was the input to the
Transformer-Encoder, consisting of $l=2$ layers with ReLU non-linearity and a
dropout of $p=0.1$. Each encoder layer uses $h=4$ MHA heads and a hidden
dimension of $C_{h}=768$. An ablation study of the Transformer-Encoder
parameters is given in Section 4.7. The Transformer-Encoder’s output $\hat{T}$
is used by the MLP regressor detailed in Table 1 that computes the quaternion
representation for the regression loss in Eq. 2. Throughout all experiments,
the model is optimized using an Adam optimizer with an initial learning rate
of $\lambda=5e-4$, with $\beta_{1}=0.9$, $\beta_{2}=0.999$,
$\epsilon=10^{-10}$, and a batch size of $20$. Our model is implemented in
PyTorch, it is end-to-end trainable, and all experiments were performed on an
8GB NVIDIA GeForce GTX 2080 GPU.

###  4.3 Comparative baselines

| InteriorNet |  | InteriorNet-T |  | SUN360 |  | StreetLearn |  | StreetLearn-T
---|---|---|---|---|---|---|---|---|---
Overlap | Method | Avg(°$\downarrow$) | Med(°$\downarrow$) | 10°(%$\uparrow$) |  | Avg(°$\downarrow$) | Med(°$\downarrow$) | 10°(%$\uparrow$) |  | Avg(°$\downarrow$) | Med(°$\downarrow$) | 10°(%$\uparrow$) |  | Avg(°$\downarrow$) | Med(°$\downarrow$) | 10°(%$\uparrow$) |  | Avg(°$\downarrow$) | Med(°$\downarrow$) | 10°(%$\uparrow$)
Large | SIFT* [33] | 6.09 | 4.00 | 84.86 |  | 7.78 | 2.95 | 55.52 |  | 5.46 | 3.88 | 93.10 |  | 5.84 | 3.16 | 91.18 |  | 18.86 | 3.13 | 22.37
SuperPoint* [13] | 5.40 | 3.53 | 87.10 |  | 5.46 | 2.79 | 65.97 |  | 4.69 | 3.18 | 92.12 |  | 6.23 | 3.61 | 91.18 |  | 6.38 | 1.79 | 16.45
Reg6D [57] | 9.05 | 5.90 | 68.49 |  | 17.00 | 11.95 | 41.79 |  | 16.51 | 12.43 | 40.39 |  | 11.70 | 8.87 | 58.24 |  | 36.71 | 24.79 | 23.03
DenseCorrVo [7] | 1.53 | 1.10 | 99.26 |  | 2.89 | 1.10 | 97.61 |  | 1.00 | 0.94 | 100.00 |  | 1.19 | 1.02 | 99.41 |  | 9.12 | 2.91 | 87.50
8PointViT [42] | 0.48 | 0.40 | 100.00 |  | 2.90 | 1.83 | 97.91 |  | - | - | - |  | 0.62 | 0.52 | 100.00 |  | 4.08 | 2.43 | 90.13
Ours | 0.45 | 0.49 | 99.55 |  | 1.88 | 0.63 | 97.95 |  | 0.57 | 0.48 | 99.95 |  | 0.60 | 0.65 | 99.3 |  | 5.45 | 1.76 | 86.50
Small | SIFT* [33] | 24.18 | 8.57 | 39.73 |  | 18.16 | 10.01 | 18.52 |  | 13.71 | 6.33 | 56.77 |  | 16.22 | 7.35 | 55.81 |  | 38.78 | 13.81 | 5.68
SuperPoint* [13] | 16.72 | 8.43 | 21.58 |  | 11.61 | 5.82 | 11.73 |  | 17.63 | 7.70 | 26.69 |  | 19.29 | 7.60 | 24.58 |  | 6.80 | 6.85 | 0.95
Reg6D [57] | 25.71 | 15.56 | 33.56 |  | 42.93 | 28.92 | 23.15 |  | 42.55 | 32.11 | 9.40 |  | 24.77 | 15.11 | 30.56 |  | 46.61 | 34.33 | 13.88
DenseCorrVol [7] | 6.45 | 1.61 | 95.89 |  | 10.24 | 1.38 | 89.81 |  | 3.09 | 1.41 | 98.50 |  | 2.32 | 1.41 | 98.67 |  | 13.04 | 3.49 | 84.23
8PointViT [42] | 1.84 | 0.94 | 99.32 |  | 4.48 | 2.38 | 96.30 |  | - | - | - |  | 1.46 | 1.09 | 100.00 |  | 9.19 | 3.25 | 87.7
Ours | 3.27 | 0.881 | 96.11 |  | 6.45 | 0.81 | 97.72 |  | 2.11 | 0.852 | 98.83 |  | 1.26 | 0.723 | 99.12 |  | 7.51 | 1.88 | 88.66
None | SIFT* [33] | 109.30 | 92.86 | 0.00 |  | 93.79 | 113.86 | 0.00 |  | 127.61 | 129.07 | 0.00 |  | 83.49 | 90.00 | 0.38 |  | 85.90 | 106.84 | 0.38
SuperPoint* [13] | 120.28 | 120.28 | 0.00 |  | – | – | 0.00 |  | 149.80 | 165.24 | 0.00 |  | – | – | 0.00 |  | – | – | 0.00
Reg6D [57] | 48.36 | 32.93 | 10.82 |  | 60.91 | 51.26 | 11.14 |  | 64.74 | 56.55 | 3.77 |  | 28.48 | 18.86 | 24.39 |  | 49.23 | 35.66 | 11.86
8PointViT [42] | - | - | - |  | - | - | - |  | - | - | - |  | - | - | - |  | - | - | -
DenseCorrVol [7] | 37.69 | 3.15 | 61.97 |  | 49.44 | 4.17 | 58.36 |  | 34.92 | 4.43 | 61.39 |  | 5.77 | 1.53 | 96.41 |  | 30.98 | 3.50 | 72.69
| Ours | 37.42 | 3.04 | 63.90 |  | 48.35 | 4.12 | 57.24 |  | 34.28 | 4.33 | 60.14 |  | 5.55 | 1.35 | 95.18 |  | 30.12 | 3.44 | 72.01

Table 2: Relative rotation estimation results. We utilized the InteriorNet,
SUN360, and StreetLearn datasets and show the average and median of geodesic
errors. We also present the percentage of image pairs with relative rotation
error below 10$\degree$, for the overlap categories in Section 4.1. The gray
numbers indicate errors exceeding $50\%$. The asterisk $*$ signifies that the
mean and median errors did not lead to pose estimation, and their calculations
are performed only on successful image pairs.

In line with Cai et al. [7], we compare our method with contemporary schemes
using the datasets in Section 4.1:

A SIFT-based approach [6]. A method for matching SIFT features [33] using
RANSAC [18] in image pairs of the same panorama, and estimating the relative
rotation matrix using Homography equations or the Essential matrix.

CNN-based methods [13, 57]. Deep learning schemes that detect and encode local
image features using SuperPointNet [13] and D2-Net [17].

Self-supervised interest point [57]. A scheme by Zhou et al. [57] (Reg6D) that
applies a CNN to approximate the mappings between various rotation
representations and fits continuous 5D and 6D rotation representations,
instead of the commonly used Euler and quaternion representations.

Extreme rotation estimation [7]. A deep learning technique to estimate the
relative 3D rotation of image pairs in an extreme setting [7] where the images
have little or no overlap. They proposed a network that automatically learns
implicit visual cues by computing a 4D correlation volume.

Attention based methods. We also compare to a recent work by Rockwell et al.
[42] (8PointVit) using a Vision Transformer (ViT) to estimate the relative
pose. Although Rockwell et al. achieve competitive results in multiple
settings, their approach is less suited for extreme view changes.

###  4.4 Experimental comparisons

Large |  |  |  |
---|---|---|---|---
Small |  |  |  |
No |  |  |  |
| StreetLearn | SUN360
Figure 4: Rotation estimation results. The panoramic images and the cropped
ground-truth images are marked in green and yellow-dot lines. The predicted
footprint of one of the cropped images is marked by the red-dot line. The
first row shows the matching results of images with large-overlaps. The second
and last rows show the matching of small-overlap and non-overlapping images.

The comparison results of the proposed scheme with baselines and SOTA schemes
are reported in Table 2. We report the mean and median of the geodesic error
given in Eq. 3 and the percentage of image pairs whose estimated relative
rotation error was under ${10^{\circ}}$. We compared the accuracy of our
proposed model to the schemes detailed in Section 4.3. The proposed approach
is shown to be accurate for both indoor and outdoor scenes and significantly
outperforms the baselines schemes in all overlap categories. For
nonoverlapping pairs, correspondence-based methods such as SIFT [6],
SuperPointNet [13], Reg6D et al. [57] and 8PointViT [42] failed to provide any
estimates, as they require feature correspondence. The DenseCorrVol approach
[7] provides accurate results in extreme cases, but our approach outperforms
it.

The qualitative results of the rotation estimation are shown in Fig. 4 for the
StreetLearn and SUN360 datasets, for the large, small, and non-overlapping
cases. We show the full panoramas, the footprints of the cropped images that
were used as inputs for the proposed scheme and the footprint of the estimated
image crop based on the estimated rotation. In all cases. we archive high
estimation accuracy.

Figure 5: Visualizations of the attention maps. The attention maps generated
by our model are overlaid on the image pairs, for the overlapping (upper) and
non-overlapping (bottom) pairs. The input crops are represented by solid
lines, and the crops are aligned with the estimated rotation as indicated by
the dotted lines.

###  4.5 Attention Maps Visualization

As described in Section 1, our scheme, together with that of Cai et al. [7],
takes advantage of rotation-informative image signals by assigning them high
attention scores. These cues can be both implicit (e.g. shadows, straight
lines, analytic shapes, lighting angles) and explicit (e.g. corners used for
feature correspondences). The visualization of these attention scores is shown
in Fig. 5, where we have overlaid the attention scores on the input images
following the approach of Kolesnikov et al. [23]. We applied our model to each
pair of images, obtained the activation maps, and superimposed them onto the
input images. The attention maps emphasize vertical and horizontal lines for
both the overlapping and non-overlapping image pairs. Our experimental
datasets (Section 4.1), consist of interior and urban outdoor scenes showing
horizontal and vertical lines. This relates to the seminal single-image camera
calibration approaches of Coughlan and Yuille [10] and Criminisi et al. [11]
and their extensions, which detect parallel lines in an image to estimate
vanishing points [20] and relative rotations [10]. Therefore, we postulate
this shows that our approach implicitly solves the rotation estimation same as
[10] and [11], by learning both the low-level informative task-specific image
cues and the higher-level algorithmic computational flow.

###  4.6 Cross-Dataset Generalization

The cross-dataset generalization properties of our approach were evaluated
using the Holicity dataset [58]. The Manhattan dataset was used to train the
models, while the London dataset was used for testing. The test images were
divided into three overlap classes according to Section 4.1. Additionally, we
compared the generalization of our approach with Cai et al.’s [7]. The
generalization results, summarized in Table 3, indicate that our approach
outperformed Cai et al.’s in all overlap classes.

Overlap | Ours [°] | Cai et al. [7] [°]
---|---|---
Large | 10.62 | 11.23
Small | 19.43 | 20.87
None | 38.65 | 40.82
Table 3: Evaluating cross-dataset generalization. We trained the models on the Manhattan dataset and tested them on the London dataset. The average geodesic error, as defined in Eq. 3, was reported. Overlap | Heads | Layers | Rotational error
---|---|---|---
Avg [°] | Med [°] | 10°[%]
Large | 1 | 1 | 1.21 | 0.97 | 99.1
4 | 1 | 0.91 | 0.78 | 99.2
2 | 2 | 0.98 | 0.83 | 99.11
4 | 2 | 0.77 | 0.65 | 99.3
4 | 4 | 0.81 | 0.69 | 99.1
Small | 1 | 1 | 8.33 | 4.13 | 93.99
4 | 1 | 7.96 | 3.51 | 94.12
2 | 2 | 6.13 | 2.12 | 95.53
4 | 2 | 4.27 | 0.881 | 95.65
4 | 4 | 4.61 | 0.98 | 95.43
None | 1 | 1 | 45.23 | 5.97 | 60.05
4 | 1 | 39.17 | 4.52 | 60.77
2 | 2 | 41.45 | 4.96 | 60.34
4 | 2 | 37.42 | 3.04 | 61.91
4 | 4 | 38.662 | 3.59 | 61.42
Table 4: Ablation study of the Transformer-Encoder parameters using the
StreeLearn dataset. For each overlap class, there exists an optimal set of
parameters that strikes a balance between the Transformer-Encoder’s expressive
capabilities (i.e., the more layers and heads, the better) and the risk of
overfitting.

###  4.7 Ablation Study

Backbone | Layers | Rotational error |
---|---|---|---
Avg [°] | Med [°] | 10°[%] |
1 | Conv(k=7, s=2, d=64) | 1.25 | 0.932 | 99.21 |
$1\times$ Residual blocks |
Conv(k=3, s=1, d=512) |
Conv(k=3, s=1, d=256) |
2 | Conv(k=7, s=2, d=64) | 0.95 | 0.76 | 99.29 |
$2\times$ Residual blocks |
Conv(k=3, s=1, d=512) |
Conv(k=3, s=1, d=256) |
3 | Conv(k=7, s=2, d=64) | 0.77 | 0.65 | 99.3 |
3 $\times$ Residual blocks |
Conv(k=3, s=1, d=512) |
Conv(k=3, s=1, d=256) |
4 | Conv(k=7, s=2, d=64) | 0.81 | 0.68 | 99.22 |
$4\times$ Residual blocks |
Conv(k=3, s=1, d=512) |
Conv(k=3, s=1, d=256) |
Table 5: Ablation of the backbone depth. We evaluate the effect of the depth
of the Residual-Unet backbone network [56], used in our scheme, by changing
the number residual blocks used.

We performed ablation studies to assess the effect of different
hyperparameters and design choices evaluated using the StreetLearn dataset.

Transformer-Encoder parameters. Table 4 summarizes multiple Transformer-
Encoder configurations for each overlap category. The expressive power of the
Transformer-Encoder depends on the number of heads and layers. The more are
used, the better the expressive power. However, using an excessive number
might lead to overfit, and the optimal constellation, in terms of accuracy in
Table 4, is given by $h=4$, $l=2$. In particular, this constellation is a
sweetspot such that increasing the number of heads or layers results in
reduced accuracy.

Representation | Loss function | Rotational error
---|---|---
Avg [°] | Med [°] | 10°[%]
Quaternions |  $L_{2}$ Regression | 0.77 | 0.65 | 99.3
Euler angles | Cross-Entropy | 1.55 | 1.322 | 99.3
Euler angles |  $L_{2}$ Regression | 1.11 | 0.98 | 99.3
Table 6: Ablation of 3D rotation encoding and training losses. We compare
encoding the 3D rotation angles by Euler angles and quaternions, in discrete
and continuous domains, and the corresponding training losses.

3D rotation encoding and training losses. The ablations of different rotation
encodings and their corresponding training losses were evaluated and are
presented in Table 6. The evaluation was performed by applying the Residual-
Unet backbone [56] and a proposed Transformer-Encoder-based cross-attention
method to image pairs from the StreeLearn dataset with large overlaps. For the
discrete formulation, in line with Cai et al. [7], the pitch and yaw angles
were discretized into 360 bins $\in$
$\left[-{180^{\circ},180^{\circ}}\right]$, and a cross-entropy loss was used
to train the network. These results were compared to those obtained using the
$L_{2}$ regression loss, as described in Eq. 2 in our scheme. The results in
Table 6 demonstrate that our $L_{2}$ regression approach outperforms the
discrete Euler angle approach proposed by Cai et al. [7].

##  5 Conclusion

We presented a novel formulation for estimating the relative rotation between
a pair of images. In particular, we study the estimation of rotations between
images with small and no overlap. We propose an attention-based approach using
a Transformer-Encoder to calculate the cross-attention between image pair
embedding maps, which outperforms the previous use of 4D correlation volumes
[49, 7]. Our framework is end-to-end trainable and optimizes a regression
loss. It has been experimentally shown to outperform previous SOTA schemes [7]
on multiple datasets used in contemporary works. in particular, for the
challenging small and nonoverlapping cases.

## References

  * [1] Hadar Averbuch-Elor and Daniel Cohen-Or.  Ringit: Ring-ordering casual photos of a temporal event.  ACM Trans. Graph., 34(3), 2015.
  * [2] Alexandru O. Balan, Michael J. Black, Horst Haussecker, and Leonid Sigal.  Shining a light on human pose: On shadows, shading and the estimation of pose and shape.  In IEEE International Conference on Computer Vision (ICCV), pages 1–8, 2007.
  * [3] Vassileios Balntas, Shuda Li, and Victor Prisacariu.  Relocnet: Continuous metric learning relocalisation using neural nets.  In European Conference on Computer Vision (ECCV), September 2018\.
  * [4] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  Surf: Speeded up robust features.  In European Conference on Computer Vision (ECCV), pages 404–417. Springer, 2006.
  * [5] Eric Brachmann, Alexander Krull, Sebastian Nowozin, Jamie Shotton, Frank Michel, Stefan Gumhold, and Carsten Rother.  Dsac-differentiable ransac for camera localization.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6684–6692, 2017.
  * [6] Matthew Brown, Richard I Hartley, and David Nistér.  Minimal solutions for panoramic stitching.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–8. IEEE, 2007.
  * [7] Ruojin Cai, Bharath Hariharan, Noah Snavely, and Hadar Averbuch-Elor.  Extreme rotation estimation using dense correlation volumes.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), pages 213–229, Cham, 2020. Springer International Publishing.
  * [9] Yaron Caspi and Michal Irani.  Aligning non-overlapping sequences.  International Journal of Computer Vision, 48(1):39–51, 2002.
  * [10] J.M. Coughlan and A.L. Yuille.  Manhattan world: compass direction from a single image by bayesian inference.  In IEEE International Conference on Computer Vision (ICCV), volume 2, pages 941–947 vol.2, 1999.
  * [11] Antonio Criminisi, Ian Reid, and Andrew Zisserman.  Single view metrology.  In IEEE International Conference on Computer Vision (ICCV), pages 434–441. IEEE Computer Society, 1999.
  * [12] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse.  Monoslam: Real-time single camera slam.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1052–1067, 2007.
  * [13] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Superpoint: Self-supervised interest point detection and description.  In IEEE Conf. Comput. Vis. Pattern Recog. Worksh., 2018.
  * [14] Mingyu Ding, Zhe Wang, Jiankai Sun, Jianping Shi, and Ping Luo.  Camnet: Coarse-to-fine retrieval for camera re-localization.  In IEEE International Conference on Computer Vision (ICCV), October 2019.
  * [15] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [16] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.  Flownet: Learning optical flow with convolutional networks.  In IEEE International Conference on Computer Vision (ICCV), pages 2758–2766, 2015.
  * [17] Mihai Dusmanu, Ignacio Rocco, Tomas Pajdla, Marc Pollefeys, Josef Sivic, Akihiko Torii, and Torsten Sattler.  D2-Net: A Trainable CNN for Joint Detection and Description of Local Features.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [18] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [19] Xiaodong Gu, Zhiwen Fan, Siyu Zhu, Zuozhuo Dai, Feitong Tan, and Ping Tan.  Cascade cost volume for high-resolution multi-view stereo and stereo matching.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495–2504, 2020.
  * [20] R. I. Hartley and A. Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * [21] Shengyu Huang, Zan Gojcic, Mikhail Usvyatsov, and Konrad Schindler Andreas Wieser.  Predator: Registration of 3D point clouds with low overlap.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [22] Di Jia, Kai Wang, ShunLi Luo, TianYu Liu, and Ying Liu.  Braft: Recurrent all-pairs field transforms for optical flow based on correlation blocks.  IEEE Signal Processing Letters, 28:1575–1579, 2021.
  * [23] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [24] S. Kong and C. Fowlkes.  Low-rank bilinear pooling for fine-grained classification.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7025–7034, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  * [25] Jinwoo Lee, Minhyuk Sung, Hyunjoon Lee, and Junho Kim.  Neural geometric parser for single image camera calibration.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), pages 541–557, Cham, 2020. Springer International Publishing.
  * [26] Jake Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, Angjoo Kanazawa, Afshin Rostamizadeh, and Ameesh Makadia.  An analysis of SVD for deep rotation estimation.  Advances in Neural Information Processing Systems (NIPS), 33, 2020\.
  * [27] Minhua Li, Qingling Chang, Yuhan Wang, Xinglin Liu, Shiting Xu, and Yan Cui.  Stereo matching with multiscale hybrid cost volume.  IEEE Access, 10:100128–100136, 2022.
  * [28] Wenbin Li, Sajad Saeedi, John McCormac, Ronald Clark, Dimos Tzoumanikas, Qing Ye, Yuzhong Huang, Rui Tang, and Stefan Leutenegger.  Interiornet: Mega-scale multi-sensor photo-realistic indoor scenes dataset.  In British Machine Vision Conference, 2018.
  * [29] Zhengfa Liang, Yulan Guo, Yiliu Feng, Wei Chen, Linbo Qiao, Li Zhou, Jianfeng Zhang, and Hengzhu Liu.  Stereo matching using multi-level cost volume and multi-scale feature constancy.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [30] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In IEEE International Conference on Computer Vision (ICCV), 2015\.
  * [31] Chenxi Liu, Alex Schwing, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler.  Rent3d: Floor-plan priors for monocular layout estimation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [32] Hai Liu, Shuai Fang, Zhaoli Zhang, Duantengchuan Li, Ke Lin, and Jiazhang Wang.  Mfdnet: Collaborative poses perception and matrix fisher distribution for head pose estimation.  IEEE Transactions on Multimedia, 24:2449–2460, 2022.
  * [33] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International Journal of Computer Vision, 60(2):91–110, 2004.
  * [34] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European Conference on Computer Vision (ECCV), 2020.
  * [35] Piotr Mirowski, Andras Banki-Horvath, Keith Anderson, Denis Teplyashin, Karl Moritz Hermann, Mateusz Malinowski, Matthew Koichi Grimes, Karen Simonyan, Koray Kavukcuoglu, Andrew Zisserman, et al.  The StreetLearn environment and dataset.  arXiv preprint arXiv:1903.01292, 2019.
  * [36] David Mohlin, Josephine Sullivan, and Gérald Bianchi.  Probabilistic orientation estimation with matrix fisher distributions.  Advances in Neural Information Processing Systems (NIPS), 33, 2020\.
  * [37] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  IEEE Transactions on Robotics, 31(5):1147–1163, 2015.
  * [38] Guang-Yu Nie, Ming-Ming Cheng, Yun Liu, Zhengfa Liang, Deng-Ping Fan, Yue Liu, and Yongtian Wang.  Multi-level context ultra-aggregation for stereo matching.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3283–3291, 2019.
  * [39] Onur Ozyesil, Vladislav Voroninski, Ronen Basri, and Amit Singer.  A survey of structure from motion.  arXiv preprint arXiv:1701.08493, 2017.
  * [40] Yiming Qian and James H. Elder.  A reliable online method for joint estimation of focal length and camera rotation.  In European Conference on Computer Vision (ECCV), page 249–265, Berlin, Heidelberg, 2022. Springer-Verlag.
  * [41] Gernot Riegler and Vladlen Koltun.  Free view synthesis.  In European Conference on Computer Vision (ECCV), 2020.
  * [42] Chris Rockwell, Justin Johnson, and David F. Fouhey.  The 8-point algorithm as an inductive bias for relative pose prediction by vits, 2022.
  * [43] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [44] Johannes L Schönberger, Marc Pollefeys, Andreas Geiger, and Torsten Sattler.  Semantic visual localization.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6896–6906, 2018.
  * [45] O. Shakil.  An efficient video alignment approach for non-overlapping sequences with free camera movement.  In ICASSP, volume 2, pages II–II, 2006.
  * [46] Che Sun, Yunde Jia, Yi Guo, and Yuwei Wu.  Global-aware registration of less-overlap rgb-d scans.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6357–6366, June 2022.
  * [47] Dekel (Basha) T., Moses Y., and Avidan S.  Photo sequencing.  International Journal of Computer Vision, 110(3):275 – 289, 2014\.  Cited by: 8.
  * [48] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7199–7209, 2018.
  * [49] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In European Conference on Computer Vision (ECCV), page 402–419, Berlin, Heidelberg, 2020. Springer-Verlag.
  * [50] M. Turkoglu, E. Brachmann, K. Schindler, G. J. Brostow, and A. Monszpart.  Visual camera re-localization using graph neural networks and relative pose supervision.  In International Conference on 3D Vision (3DV), pages 145–155, Los Alamitos, CA, USA, dec 2021.
  * [51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017.
  * [52] Jianxiong Xiao, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Recognizing scene viewpoint using panoramic place representation.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2695–2702. IEEE, 2012.
  * [53] Fei Xue, Xin Wu, Shaojun Cai, and Junqiu Wang.  Learning multi-view camera relocalization with graph neural networks.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11372–11381, 2020.
  * [54] Gengshan Yang and Deva Ramanan.  Volumetric correspondence networks for optical flow.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems (NIPS), volume 32. Curran Associates, Inc., 2019\.
  * [55] Zhengxin Zhang, Qingjie Liu, and Yunhong Wang.  Road extraction by deep residual u-net.  IEEE Geoscience and Remote Sensing Letters, 15:749–753, 2018.
  * [56] Zhengxin Zhang, Qingjie Liu, and Yunhong Wang.  Road extraction by deep residual u-net.  IEEE Geoscience and Remote Sensing Letters, 15(5):749–753, 2018\.
  * [57] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5745–5753, 2019.
  * [58] Yichao Zhou, Jingwei Huang, Xili Dai, Linjie Luo, Zhili Chen, and Yi Ma.  HoliCity: A city-scale data platform for learning holistic 3D structures.  2020\.  arXiv:2008.03286 [cs.CV].
  * [59] Rui Zhu, Xingyi Yang, Yannick Hold-Geoffroy, Federico Perazzi, Jonathan Eisenmann, Kalyan Sunkavalli, and Manmohan Chandraker.  Single view metrology in the wild.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), pages 316–333, Cham, 2020. Springer International Publishing.