# GPR-Net: Multi-view Layout Estimation via a Geometry-aware Panorama
Registration Network

Jheng-Wei Su1, Chi-Han Peng2, Peter Wonka3, Hung-Kuo Chu4
1,4 National Tsing Hua University
2 National Yang Ming Chiao Tung University
3 King Abdullah University of Science and Technology (KAUST)
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Reconstructing 3D layouts from multiple $360^{\circ}$ panoramas has received
increasing attention recently as estimating a complete layout of a large-scale
and complex room from a single panorama is very difficult. The state-of-the-
art method, called PSMNet (Wang et al., 2022), introduces the first learning-
based framework that jointly estimates the room layout and registration given
a pair of panoramas. However, PSMNet relies on an approximate (i.e., ”noisy”)
registration as input. Obtaining this input requires a solution for _wide
baseline registration_ which is a challenging problem. In this work, we
present a complete multi-view panoramic layout estimation framework that
jointly learns panorama registration and layout estimation given a pair of
panoramas without relying on a pose prior. The major improvement over PSMNet
comes from a novel Geometry-aware Panorama Registration Network or GPR-Net
that effectively tackles the wide baseline registration problem by exploiting
the layout geometry and computing fine-grained correspondences on the layout
boundaries, instead of the global pixel-space. Our architecture consists of
two parts. First, given two panoramas, we adopt a vision transformer to learn
a set of 1D horizon features sampled on the panorama. These 1D horizon
features encode the depths of individual layout boundary samples and the
correspondence and covisibility maps between layout boundaries. We then
exploit a non-linear registration module to convert these 1D horizon features
into a set of corresponding 2D boundary points on the layout. Finally, we
estimate the final relative camera pose via RANSAC and obtain the complete
layout simply by taking the union of registered layouts. Experimental results
indicate that our method achieves state-of-the-art performance in both
panorama registration and layout estimation on a large-scale indoor panorama
dataset ZInD (Cruz et al., 2021).

##  1 Introduction

In this paper, we tackle the problem of room layout estimation from multiple
$360^{\circ}$ panoramas. Many approaches that can estimate room layouts from a
single panorama have been proposed (Zou et al., 2018; Yang et al., 2019; Sun
et al., 2019; Pintore et al., 2020). However, these methods did not take
advantage of ”multi-view” data in which multiple panoramas are taken to better
capture a single room. These kinds of data are actually common as evidenced by
several indoor datasets such as ZInD (Cruz et al., 2021), Matterport3D (Chang
et al., 2017), Gibson (Xia et al., 2018), and Structure3D (Zheng et al., 2020)
in which photographers often take multiple panoramas to better capture
complex, non-convex rooms that would be partially occluded from just a single
location.

Our work mainly improves upon a recent paper, PSMNet (Wang et al., 2022), that
tackled the problem of layout estimation from two panoramas captured in the
same room. The idea of PSMNet is to build an architecture that first registers
two panoramas in their ceiling view projections and then jointly estimates a
2D layout segmentation. An important aspect of their architecture is that the
layout estimation and registration can be trained jointly. However, a major
limitation of PSMNet (also mentioned in their paper) is that the architecture
relies on an initial approximate registration. The authors argued that such an
approximate registration could be given either manually or computed by
external methods such as Structure from Motion (SfM) methods or Shabani et al.
(2021). While a manual registration may work, the method would no longer be
automatic. When experimenting with existing methods for approximate
registration, we observed that they frequently make registration errors and
even fail to provide a registration in a substantial number of cases. The main
reason is that the required registration mainly falls into the category of
wide baseline registration with only two given images. For example, our
results show that the state-of-the-art SfM method OpenMVG (Moulon et al.,
2016) fails to register $76\%$ of panorama pairs from our test dataset. It is
thus impractical to assume an independent algorithm that can reliably give an
approximate solution to the challenging wide baseline registration problem. In
addition, relying on such an algorithm moves a critical part of the problem to
a pre-process.

Therefore, we set out to develop a complete multi-view panorama registration
and layout estimation framework that no longer relies on an approximate
registration given as input as shown in Figure 1. To achieve this, we propose
a novel Geometry-aware Panorama Registration Network, or GPR-Net, based on the
following design ideas. First, our experiments indicate that a global (pixel-
space) registration that directly regresses pose parameters (i.e., translation
and rotation) is too ambitious. Instead, we propose to compute more fine-
grained correspondences in a different space. Specifically, GPR-Net
conceptually samples the layout boundaries of two input layouts and computes
features for the sampled locations. For each boundary sample in each of the
two panoramas, it estimates the distance from the camera (depth). In addition,
it estimates the correspondence map from the samples in the first panorama to
the second panorama and a covisibility map describing if a sample in the first
panorama is visible in the second panorama. Each of these maps (depth,
correspondence, and covisibility) is a 1D sequence of values.

This representation has the advantages of having more elements to register
(e.g., 256 samples per panorama) and more supervision signal for fine-grained
estimation. This leads to better learning performance. Second, we build a non-
linear registration module to compute the final relative camera pose. The
module combines two horizon-depth maps with the horizon-correspondence and
horizon-covisibility maps to obtain a set of covisible corresponding boundary
samples in a 2D coordinate system aligned with the ceiling plane, followed by
a RANSAC-based pose estimation. Note that this non-linear space is more
expressive and can encode a richer range of maps between two panoramas. The
final complete layout is obtained simply by taking the union of two registered
layouts.

We extensively validate our model by comparing with the state-of-the-art
panorama registration method and multi-view layout estimation method on a
large-scale indoor panorama dataset ZInD (Cruz et al., 2021). The experimental
results demonstrate that our model is superior to competing methods by
achieving a significant performance boost in both panorama registration
accuracy (mAA$@5^{\circ}$: $+68.5\%$(rotation), $+63.0\%$(translation),
mAA$@10^{\circ}$: $+74.1\%$(rotation), $+72.3\%$(translation)) and layout
reconstruction accuracy (2D IoU $+4.5\%$).

\begin{overpic}[width=397.48499pt]{Figures/images/Overall_Pipeline_v2.png}
\par\end{overpic}
Figure 1: Proposed multi-view panorama registration and layout estimation
framework. Given two panorama images, a neural network (GPR-Net) jointly
predicts layout boundary correspondences and individual layouts for the two
panoramas (section 3.1,section 3.2). The correspondences are fed to a
registration module to compute the relative camera pose $(T,R)$ (section 3.3).
A layout fusion module then computes a unified 3D layout given the camera pose
and the individual layouts (section 3.4).

In summary, our contributions are as follows:

  * •

We propose the first complete multi-view panoramic layout estimation
framework. Our architecture jointly learns the layout and registration from
data, is end-to-end trainable, and most importantly, does not rely on a pose
prior.

  * •

We devise a novel panorama registration framework to effectively tackle the
wide baseline registration problem by exploiting the layout geometry and
computing a fine-grained correspondence of samples on the layout boundaries.

  * •

We achieve state-of-the-art performance on ZInD (Cruz et al., 2021) dataset
for both the stereo panorama registration and layout reconstruction tasks.

##  2 Related Work

###  2.1 Single-view room layout estimation

There exist many methods to estimate the room layouts from just a single image
taken inside an indoor environment. Methods that take only one perspective
image include earlier attempts that relied on image clues and optimization
(Hedau et al., 2009; Hoiem et al., 2007; Ramalingam & Brand, 2013) and later
neural networks (Lee et al., 2017; Yan et al., 2020). Capturing the increasing
availability and popularity of full $360^{\circ}$ panoramic images, the
seminal work by Zhang et al. (Zhang et al., 2014) proposed to take panoramas
as native inputs for scene understanding. Recently, several methods were
proposed to predict the room layouts from a single panorama using neural
networks. A major difference between these methods is the assumption on the
shape of the room layouts - from being strictly a cuboid (Zou et al., 2018),
Manhattan world (Yang et al., 2019; Sun et al., 2019), to more recently
general 2D layouts (Atlanta world) (Pintore et al., 2020). For our work, we
choose to adopt the Manhattan assumption because more corresponding data is
available. See Zou et al. (2021) for a thorough survey on predicting Manhattan
room layouts from a single panorama. More recent methods delivered state-of-
the-art performance by transforming the problem into a depth-estimation one
(Wang et al., 2022) or by leveraging powerful transformer-based network
architecture (Jiang et al., 2022). Although these single-view methods perform
well in the cuboid and L-shape rooms, they tend to fail in the large-scale,
complex and non-convex rooms where a single-view panorama covers only part of
the whole space due to occlusion.

###  2.2 Panorama registration

Image registration, i.e., finding transformations between the cameras of two
or multiple images taken of the same scene, is a key component of Structure-
from-Motion (SfM). See Özyeşil et al. (2017) for a recent survey and Hartley &
Zisserman (2003) for an extensive study. Registration problems can be
categorized by: 1) the assumptions about the camera model, e.g., perspective
(pinhole camera), weak-perspective, or orthographic, 2) the assumptions about
the transformation, e.g., rigid, affine, or general non-rigid, and 3) the
types of the image inputs, e.g., perspective images or full $360^{\circ}$
panoramas, and with/without depths. In addition, the difficulty differs
greatly on whether the images are taken densely or sparsely. Modern takes on
registration problems often leverage state-of-art programs/libraries such as
COLMAP (Schönberger & Frahm, 2016) and OpenMVG (Moulon et al., 2016). Our
problem falls into a lesser-studied category: registering rigid transforms
between sparse panoramas. While there exist methods that tackle sparse
perspective image inputs (Salaün et al., 2017; Fabbri et al., 2020) and
methods that handle panoramas natively (Pagani & Stricker, 2011; Taneja et
al., 2012; Ji et al., 2020), our results show that we can improve upon the
state-of-the-art panorama registration methods in our sparse view setting. A
key bottleneck is that traditional SfM methods fail to handle the wide
baseline registration problem where the views are far apart from each other.
In Shabani et al. (2021), SfM of extremely sparse panoramas was tackled by
matching room types and specific elements such as doors and windows. In
contrast to previous methods that perform the registration in the global
pixel-space, we propose a novel learning-based panorama registration framework
that directly compute the registration between two panoramas without taking
any prior knowledge as input. Our method may have some similarities to the
ECCV 2022 paper (Hutchcroft et al., 2022), but the paper was not available
online when we developed our method or wrote the first drafts of our paper.

###  2.3 Scene reconstruction using sparse panoramas

Attempts to reconstruct indoor scenes using just a handful of RGB panoramas as
inputs Pintore et al. (2018; 2019) are nascent but promising since
photographers are adapting $360^{\circ}$ cameras into their workflows (e.g.,
Matterport 3D capture system (Chang et al., 2017)) and it is awkward to
capture dense panoramic inputs due to camera/tripod setups. While previous
methods assume that all the input panoramas are already registered, the PSMNet
(Wang et al., 2022) introduces the first learning-based framework that jointly
estimates the room layout and registration given a pair of panoramas. However,
it still has a major bottleneck that an initial approximate (noisy)
registration must be given (e.g., either manually specified or computed by
external methods) during both the training and inference stages. Our GPR-Net
is also an end-to-end deep neural network that jointly learns the room layout
and panorama registration. Most importantly, our model does not rely on a pose
prior and is thus suitable for real-world application scenarios.

##  3 Methodology

\begin{overpic}[width=397.48499pt]{Figures/images/architecture_woconcat.png}
\par\end{overpic}
Figure 2: GPR-Net architecture. Our network architecture follows the standard
vision transformer with an encoder-decoder scheme and multiple MLP heads.
Given a pair of panoramas ${\mathbf{I^{1}}}$ and ${\mathbf{I^{2}}}$, our
network first extracts feature maps as input tokens by ResNet-50 and feeds
those features into the transformer ${\mathcal{F}^{T}}$. We then treat the
output tokens as boundary samples ${\mathbb{U}}$ in panorama
${\mathbf{I^{1}}}$ and process the output of the transformer
${\mathcal{F}^{T}}$ by different MLP heads for horizon ceiling/floor
coordinates, namely ${\mathbb{V}^{j=(c,f)}_{k=(1,2)}}$, horizon-covisibility
map ${\mathbb{C}}$, and horizon-correspondence map ${\mathbb{O}}$,
respectively. We perform L2D transformation on horizon ceiling/floor
coordinates ${\mathbb{V}^{j=(c,f)}_{k=(1,2)}}$ to obtain horizon-depth map
$\mathbb{D}$. Finally, we compute the layout loss $\mathcal{L}_{layout}$,
covisibility loss $\mathcal{L}_{covis}$, correspondence loss
$\mathcal{L}_{cor}$, and cycle-consistentcy loss $\mathcal{L}_{cycle}$ using
the corresponding 1D output horizon maps.

###  3.1 Network architecture

Figure 2 illustrates the GPR-Net architecture. GPR-Net uses building blocks
from COTR (Jiang et al., 2021) and LED2-Net (Wang et al., 2021). First, we
feed two (vertically) axis aligned panoramas ${\mathbf{I^{1}}}$ and
${\mathbf{I^{2}}}$ into a ResNet-50 (He et al., 2015) feature extractor and
generate two feature maps of resolution $16\times 8$. Following the encoder-
decoder transformer architecture, we feed the extracted feature maps into the
transformer encoder block using the two sets of $16\times 8$ pixels as input
tokens. The output tokens of the transformer encoder block will be used for
cross attention in the transformer decoder block. We use a 2D $UV$ coordinate
system to parametrize a panorama image with $(u,v)$ in $[0,1]\times[-1,1]$.
The $u$ coordinate describes the horizontal position and the $v$ coordinate
the vertical position. We want to query multiple values (height, depth,
correspondences, covisibility) for different $u$ coordinates. We therefore
uniformly sample the $u$ coordinate $\in[0,1]$ with $N$ evenly distributed
samples. We obtain the set ${\mathbb{U}}=\\{u_{i}\\}_{i=1}^{N}$. The query
samples are encoded by linear positional encoding and are the input tokens of
the transformer decoder. The output tokens of the transformer decoder encode
multiple types of information about the boundary samples. To extract this
information, the output tokens are further processed using multiple MLP heads.
Specifically, we use: (i) _layout MLP heads_ ${\mathcal{F}^{c}_{k{}}}$ and
${\mathcal{F}^{f}_{k{}}}$ with $k=(1,2)$. The outputs of
${\mathcal{F}^{c}_{k{}}}$ and ${\mathcal{F}^{f}_{k{}}}$ are the $v$
coordinates of the ceiling and floor boundaries in the panorama images,
respectively. We denote the $v$ coordinates of the ceiling boundaries as
${\mathbb{V}^{c}_{k=(1,2)}}$ and the floor boundaries,
${\mathbb{V}^{f}_{k=(1,2)}}$. For each ${\mathbb{V}^{j=(c,f)}_{k=(1,2)}}$, we
further exploit the Layout-to-Depth (L2D) transformation (Wang et al., 2021)
to generate a corresponding horizon-depth map
${\mathbb{D}^{j=(c,f)}_{k=(1,2)}}$; (ii) _correspondence MLP head_
${\mathcal{F}^{cor}}$ that outputs a horizon-correspondence map
${\mathbb{O}}=\\{o_{i}\\}_{i=1}^{N}$, where $o_{i}$ indicates the
correspondence between $u_{i}\in{\mathbf{I^{1}}}$ and
$o_{i}\in{\mathbf{I^{2}}}$; and (iii) _covisibility MLP head_
${\mathcal{F}^{covis}}$ that outputs a horizon-covisibility map
${\mathbb{C}}=\\{c_{i}\\}_{i=1}^{N}$, where $c_{i}$ is a value $\in[0,1]$,
encoding whether the $i-th$ element in ${\mathbb{O}}$ should be considered
($c_{i}=1$) or not ($c_{i}=0$) in the pose estimation.

###  3.2 Loss functions

Here we elaborate on the layout, correspondence, covisibility, and cycle-
consistency loss functions used for training our network.

Layout loss calculates the low-level geometry loss between the predicted
horizon-depth maps ${\mathbb{D}}$ of input panoramas ${\mathbf{I^{1}}}$ and
${\mathbf{I^{2}}}$ and their ground-truth horizon-depth maps
${\mathbb{\overline{D}}}$. The loss is defined as follows:

$\mathcal{L}_{layout}=\frac{1}{M}\sum_{j=(c,f)}\sum_{k=(1,2)}\|{\mathbb{D}^{j}_{k}}-{\mathbb{\overline{D}}^{j}_{k}}\|_{1},$
(1)

where $M$ is the dimension of the horizon-depth maps.

Covisibility loss evaluates the predicted normalized horizon co-visibility map
${\mathbb{C}}=\\{c_{i}\\}_{i=1}^{N}$ with respect to the ground-truth horizon
co-visibility map $\mathbb{\overline{C}}=\\{\overline{c}_{i}\\}_{i=1}^{N}$.
The loss is defined as follows:

$\mathcal{L}_{covis}=\frac{1}{N}\sum_{i=1}^{N}\alpha\overline{c}_{i}\cdot\log(c_{i})+(1-\overline{c}_{i})\cdot\log(1-c_{i}),$
(2)

where $\alpha$ is the hyperparameter for weighting the positive samples. We
use $\alpha$ because there are a lot more positive than negative samples in
the ground truth.

Correspondence loss calculates the difference between the predicted horizon-
correspondence map ${\mathbb{O}}=\\{o_{i}\\}_{i=1}^{N}$ and the ground-truth
horizon-correspondence map
$\mathbb{\overline{O}}=\\{\overline{o}_{i}\\}_{i=1}^{N}$. The loss is defined
as follows:

$\mathcal{L}_{cor}=\frac{1}{N}\sum_{i=1}^{N}\left\\{\begin{array}[]{rcl}\min(\|o_{i}-\overline{o}_{i}\|_{1},\|1-\overline{o}_{i}+o_{i}\|_{1}),&\text{if
}\overline{c}_{i}\geq 0.5\\\ 0,&\text{otherwise}\end{array}\right.,$ (3)

where we use a cyclic loss instead of the simple L1 loss between the predicted
and ground-truth correspondence to adopt the coordinate system of in
equirectangular projection.

Cycle-consistency loss (Jiang et al., 2021) enforces the network outputs to be
cycle-consistent and adapts the network to different ray casting positions in
contrast to the uniformly sampled ray casting positions. We reverse the order
of the two panoramas ${\mathbf{I^{1}}}$ and ${\mathbf{I^{2}}}$, and treat the
ground-truth horizon-correspondence map $\\{\overline{o}_{i}\\}_{i=1}^{N}$ as
input and the original input $\\{o_{i}\\}_{i=1}^{N}$ as target correspondence.
We separate the cycle-consistency loss into two parts as follows:

$\mathcal{L}^{cor}_{cycle}=\frac{1}{N}\sum_{i=1}^{N}\left\\{\begin{array}[]{rcl}\min(\|{\mathcal{F}^{cor}}({\mathcal{F}^{T}}(\overline{o}_{i}))-u_{i}\|_{1},\|1-u_{i}+{\mathcal{F}^{cor}}({\mathcal{F}^{T}}(\overline{o}_{i}))\|_{1}),&\text{if
}\overline{c}_{i}\geq 0.5\\\ 0,&\text{otherwise}\end{array}\right.$ (4)

and

$\mathcal{L}^{covis}_{cycle}=\frac{1}{N}\sum_{i=1}^{N}\alpha\overline{c}_{i}\cdot\log({\mathcal{F}^{covis}}({\mathcal{F}^{T}}(\overline{o}_{i})))+(1-\overline{c}_{i})\cdot\log(1-{\mathcal{F}^{covis}}({\mathcal{F}^{T}}(\overline{o}_{i}))).$
(5)

Here, we still use the ground-truth horizon-covisibility map
$\\{\overline{c}_{i}\\}_{i=1}^{N}$ as the target in
$\mathcal{L}^{covis}_{cycle}$ since the order of
$\\{\overline{o}_{i}\\}_{i=1}^{N}$ and $\\{o_{i}\\}_{i=1}^{N}$ is the same.

Finally, the overall loss function used in our network is defined as follows:

$\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{layout}+\lambda_{2}\mathcal{L}_{cor}+\lambda_{3}\mathcal{L}_{covis}+\lambda_{4}\mathcal{L}^{cor}_{cycle}+\lambda_{5}\mathcal{L}^{covis}_{cycle},$
(6)

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$, $\lambda_{4}$, and
$\lambda_{5}$ are the hyperparameters for weighting the loss functions.

\begin{overpic}[width=397.48499pt]{Figures/images/Registration_wray.png}
\par\end{overpic}
Figure 3: Registration pipeline. We start our registration process by (a)
utilizing the elementary geometric transformations (Wang et al., 2021) to
obtain: 1) two horizon-depth maps ${\mathbb{D}^{c}_{1}}$ and
${\mathbb{D}^{c}_{2}}$, and 2) two 2D point sets ${\mathbb{P}^{1}}$ and
${\mathbb{P}^{2}}$. (b) Considering the non-uniformly distributed horizon-
correspondence map ${\mathbb{O}}$, we extract the one-to-one point-wise
correspondence ${\mathbb{P}^{1}}$ and ${\mathbb{\hat{P}}^{1}}$ by
interpolating ${\mathbb{P}^{2}}$ with ${\mathbb{O}}$. Finally, (c) we filter
out some of the in-covisible pairs via horizon co-visibility map
${\mathbb{C}}$ and (d) robustly estimate final pose parameters by RANSAC.

###  3.3 Non-linear registration

Figure 3 illustrates the pipeline of our non-linear registration. We use a 3D
Cartesian coordinate system to perform our registration process where the
$y$-axis is the up axis, the camera center is the origin, and the $XZ$-plane
is parallel to floor and ceiling. We reuse the elementary geometric
transformations described by Wang et al. (2021) to obtain the following: 1)
the horizon-depth maps
${\mathbb{D}^{c}_{1}}=\\{d_{i}^{1}\in\mathbb{R}^{1}\\}_{i=1}^{M}$ and
${\mathbb{D}^{c}_{2}}=\\{d_{i}^{2}\in\mathbb{R}^{1}\\}_{i=1}^{M}$, which are
derived from the predicted ceiling layout boundaries ${\mathbb{V}^{c}}$ and
represent the $M$ evenly sampled distances from the origin to the predicted
layout boundary in the $XZ$-plane; 2) two sets of 2D points
${\mathbb{P}^{1}}=\\{p_{i}^{1}\in\mathbb{R}^{2}\\}_{i=1}^{M}$ and
${\mathbb{P}^{2}}=\\{p_{i}^{2}\in\mathbb{R}^{2}\\}_{i=1}^{M}$. These points
are on the layout boundary on the $XZ$-plane. Each point corresponds to a
depth value in ${\mathbb{D}^{c}_{1}}$ or ${\mathbb{D}^{c}_{2}}$, respectively
(Figure 3(a)). We would like to remark that we only consider 3-DoF
transformations in ZInD (Cruz et al., 2021) following (Wang et al., 2022) and
that the conversion of Wang et al. (2021) includes a resampling step from $N$
to $M$ boundary samples. In order to compute a one-to-one point-wise
correspondence between ${\mathbb{P}^{1}}$ and ${\mathbb{P}^{2}}$, we exploit
the estimated horizon-correspondence map ${\mathbb{O}}$. Since ${\mathbb{O}}$
is not necessary a uniform distribution, for each
$p_{i}^{1}\in{\mathbb{P}^{1}}$, we compute its corresponding point via
interpolating ${\mathbb{P}^{2}}$ with ${\mathbb{O}}$ (Figure 3(b)). We denote
the corresponding boundary points as
${\mathbb{\hat{P}}^{1}}=\\{\hat{p}_{i}^{1}\\}_{i=1}^{M}$. We then filter out
the matched pair in ${\mathbb{P}^{1}}$ and ${\mathbb{\hat{P}}^{1}}$ according
to the horizon co-visibility map ${\mathbb{C}}$ (Figure 3(c)). The final pose
parameters (i.e., translation $T$ and rotation $R$) are computed via a RANSAC-
based estimation (Figure 3(d)).

###  3.4 Layout fusion

Given the relative camera pose between input panoramas, we combine two
individual partial layouts into a unified one as follows. First, for each
input panorama, we adopt the same post-processing procedure as LED2-Net (Wang
et al., 2021) that converts the estimated horizon-ceiling coordinates
${\mathbb{V}^{c}_{k}}$ and horizon-floor coordinates ${\mathbb{V}^{f}_{k}}$
into a 2D layout map $L_{k}$ and a layout height $H_{k}$. Then, we register
two 2D layout maps using the estimated relative camera pose and then combine
them into a complete 2D layout map via a union operation $L_{final}=L_{1}\cup
L_{2}$. The final 3D layout is obtained by extruding $L_{final}$ with the
average layout height $(H_{1}+H_{2})/2$.

##  4 Results

In this section, we compare our method with state-of-the-art layout
reconstruction and panorama registration approaches. We also conducted
multiple ablation studies to validate the necessity of individual modules in
our architecture.

###  4.1 Experimental Settings

Dataset. We conduct all the experiments on a public indoor panorama dataset,
Zillow Indoor Dataset (ZInD), which contains 67,448 indoor panoramas. We
follow the same procedure as PSMNet to select the panorama pair instances and
obtain training (105256), validation (12376), and tests (12918) pairs. While
we tried to match the PSMNet test protocol as closely as possible and
exchanges multiple emails with the authors to that effect, we are still
waiting for the authors of PSMNet to release their testing code and dataset
split.

Competing methods. We compare our method with the following state-of-the-art
layout reconstruction models, LED2-Net (Wang et al., 2021), LGT-Net (Jiang et
al., 2022), and PSMNet (Wang et al., 2022). Since LED2-Net and LGT-Net are
single-view layout estimation methods, we first estimate the layout for each
view, register two input panoramas using OpenMVG, and then perform a union
operation to obtain the final reslut. Note that in cases where OpenMVG fails
to produce a registration, we use average ground-truth pose of the training
datasets. To evaluate the performance on the panorama registration, we compare
our GPR-Net with OpenMVG, a popular Structure-from-Motion library that
supports stereo panorama matching. We followed the official settings for the
feature extractor and correspondence matching and applied ’incrementalv2’ mode
for SfM operation to achieve the best reconstruction rate.

Evaluation metrics. We used 2D IoU, 3D IoU, and $\delta^{i}$ for quantitative
evaluation of the layout reconstruction. As for measuring the image
registration quality, we used angular error between estimated relative pose
and ground-truth in translation($\Delta t$) and rotation($\Delta R$)
(Brachmann & Rother, 2019) and mean-average-accuracy (Jiang et al., 2021) of
translation and rotation at a $5^{\circ}$ and $10^{\circ}$ error threshold,
denoted as T-mAA$@5^{\circ}$, T-mAA$@10^{\circ}$, R-mAA$@5^{\circ}$, and
R-mAA$@10^{\circ}$, respectively.

Implementation details. We implemented our model in PyTorch and conducted
experiments on a single NVIDIA V100 with 32GB VRAM. The resolution of the
panoramas is resized to $512\times 256$. We use the Adam optimizer with b1=0.9
and b2=0.999. The learning rates of the transformer and the ResNet-50 are 1e-4
and 1e-5, and the batch size is set to 8. We empirically set
$\lambda_{1}=1,\lambda_{2}=1,\lambda_{3}=1,\lambda_{4}=1,\lambda_{5}=1$ in
Equation 6, $\alpha=0.1$ in Equation 2, $M=256$ in Equation 1, and $N=256$ in
Section 3.2.

###  4.2 Evaluation on layout reconstruction

Table 1: Quantitative comparisons with state-of-the-art methods on layout reconstruction. The * symbol means that the numbers are reported in PSMNet Wang et al. (2022).  | w/ GT pose | w/o GT pose
---|---|---
Method | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$ | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$
LED2Net (Wang et al., 2021) | 0.8364 | 0.9557 | 0.8131 | 0.5889 | 0.8777 | 0.5738
LGT-Net (Jiang et al., 2022) | 0.8388 | 0.9537 | 0.8126 | 0.5831 | 0.8779 | 0.5661
PSMNet* (Wang et al., 2022) | 0.8101 | 0.9238 | - | 0.7577 | 0.9217 | -
GPR-Net (Ours) | 0.8449 | 0.9603 | 0.8211 | 0.8026 | 0.9452 | 0.7816
Table 2: Quantitative comparisons with state-of-the-arts methods on indoor
panorama registration. We evaluate all the methods on the ZInD dataset with
our testing set.

| SfM successful pair | full testing pair
---|---|---
Method | $\Delta t\downarrow$ | $\Delta R\downarrow$ | R-mAA$@5^{\circ}\uparrow$ | R-mAA$@10^{\circ}\uparrow$ | T-mAA$@5^{\circ}\uparrow$ | T-mAA$@10^{\circ}\uparrow$
OpenMVG (Moulon et al., 2016) | 17.6717 | 7.3707 | 0.1787 | 0.1969 | 0.1369 | 0.1627
GPR-Net (Ours) | 6.9907 | 2.5514 | 0.8633 | 0.9378 | 0.7667 | 0.8854

In this experiment, we evaluate both the qualitative and quantitative
performance of our model on the layout reconstruction task by comparing with
baselines. The qualitative results are shown in Figure 4. In short, our method
produces more accurate layout reconstruction than LED2-Net and LGT-Net. Note
that visual comparison with PSMNet is infeasible since the relevant code and
data are not released. Our method also achieves the best performance against
all the baselines across all evaluation metrics, as shown in Table 1. In the
setting without ground-truth (GT) camera poses, our model shows an improvement
over the PSMNet by $4.49\%$ for 2D IoU and 0.02 for $\delta^{i}$ without
needing a (noisy) pose prior. Please refer to the supplementary material for
more visual comparisons.

Input

Ours

LED2-Net

LGT-Net

Figure 4: Visual comparisons. We show visual comparisons with other competing
methods categorized by difficulty. From top to bottom, we select cases where
our reconstruction accuracy is the range of top 10%, 20%, 50%, and bottom 10%
in our test set. The first column shows two input panoramas with their
estimated layouts. The remaining columns show the ground-truth layout, our
layouts, LED2-Net’s layouts, and LGT-Net’s layouts in blue, green, yellow,
red, respectively.

###  4.3 Evaluation on indoor panorama registration

In this experiment, we compare GPR-Net with OpenMVG in terms of image
registration quality on the indoor panoramas. As shown in Table 2, GPR-Net
outperforms OpenMVG across all evaluation metrics. The main problem of OpenMVG
is that it has many failure cases where the algorithm returns no registration.
Therefore, we show a separated comparison on the subset of panorama pairs
where OpenMVG successfully returns a result. Note that even in these cases,
our model beats OpenMVG with a noticeable improvement in angular errors.
Moreover, our model overwhelms OpenMVG when evaluating how many pairs are
successfully registered within a certain error threshold.

###  4.4 Ablation Study

We conducted ablation studies to validate our method from different
perspectives.

The influence of the number of cast rays. In this experiment, we start with
the default setting of $N=256$ sample points along the $u$ coordinate and
progressively add more samples during the inference without re-training the
network. As shown in Table 4 and Table 3, we obtain the best performance with
the default setting on both the layout reconstruction and panorama
registration. We conclude that re-training the network would be necessary for
increasing the number of samples, but this will significantly increase
computation time.

Table 3: Number of samples $N$ vs. registration accuracy.

| SfM successful pair | full testing pair
---|---|---
$N$ | $\Delta t\downarrow$ | $\Delta R\downarrow$ | R-mAA$@5^{\circ}\uparrow$ | R-mAA$@10^{\circ}\uparrow$ | T-mAA$@5^{\circ}\uparrow$ | T-mAA$@10^{\circ}\uparrow$
256 | 6.9907 | 2.5514 | 0.8633 | 0.9378 | 0.7667 | 0.8854
512 | 16.9696 | 8.3937 | 0.6390 | 0.7183 | 0.5171 | 0.6416
1024 | 16.5873 | 7.2243 | 0.6825 | 0.7314 | 0.5901 | 0.6701

Table 4: Number of samples $N$ vs. layout reconstruction accuracy. | w/ GT pose | w/o GT pose
---|---|---
$N$ | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$ | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$
256 | 0.8449 | 0.9603 | 0.8211 | 0.8026 | 0.9452 | 0.7816
512 | 0.7853 | 0.9472 | 0.7633 | 0.6917 | 0.8618 | 0.6730
1024 | 0.7619 | 0.9405 | 0.7407 | 0.6569 | 0.8437 | 0.6392

The effect of joint optimization architecture. In this experiment, we divided
the layout prediction and the correspondence prediction into two individual
models. Specifically, we use LED2-Net as our layout prediction model and the
vision transformer as our correspondence prediction model. As shown in Table
5, we obtain better accuracy using our joint prediction model on all the
layout reconstruction metrics.

Table 5: Joint optimize architecture vs. layout reconstruction accuracy. | w/ GT pose | w/o GT pose
---|---|---
Method | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$ | 2D IoU$\uparrow$ |  $\delta^{i}$ $\uparrow$ | 3D IoU$\uparrow$
w/o joint optimization | 0.8364 | 0.9557 | 0.8131 | 0.7920 | 0.9413 | 0.7713
w/ joint optimization | 0.8449 | 0.9603 | 0.8211 | 0.8026 | 0.9452 | 0.7816

##  5 Conclusions

We present a first complete solution for room layout reconstruction from a
pair of panorama images. In contrast to previous work, i.e. PSMNet, we do not
rely on an approximate registration but can register the two panorama images
directly. The major improvement over PSMNet comes from a novel Geometry-aware
Panorama Registration Network (GPR-Net) that effectively tackles the wide
baseline registration problem. We propose to exploit the layout geometry and
compute fine-grained correspondences between the two layout boundaries, rather
than directly computing the registration on global pixel-space. The main
limitation of our method is that the layout fusion block that processes two
layouts is very simple. We recommend the development of learned fusion modules
as major avenue for future work.

## References

  * Brachmann & Rother (2019) Eric Brachmann and Carsten Rother.  Neural-guided ransac: Learning where to sample model hypotheses.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Chang et al. (2017) Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _International Conference on 3D Vision (3DV)_ , 2017.
  * Cruz et al. (2021) Steve Cruz, Will Hutchcroft, Yuguang Li, Naji Khosravan, Ivaylo Boyadzhiev, and Sing Bing Kang.  Zillow indoor dataset: Annotated floor plans with 360deg panoramas and 3d room layouts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2133–2143, June 2021.
  * Fabbri et al. (2020) Ricardo Fabbri, Timothy Duff, Hongyi Fan, Margaret H. Regan, David da Costa de Pinho, Elias Tsigaridas, Charles W. Wampler, Jonathan D. Hauenstein, Peter J. Giblin, Benjamin Kimia, Anton Leykin, and Tomas Pajdla.  Trplp - trifocal relative pose from lines at points.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Hartley & Zisserman (2003) Richard Hartley and Andrew Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, USA, 2 edition, 2003.  ISBN 0521540518.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _arXiv preprint arXiv:1512.03385_ , 2015.
  * Hedau et al. (2009) V. Hedau, D. Hoiem, and D. Forsyth.  Recovering the spatial layout of cluttered rooms.  In _2009 IEEE 12th International Conference on Computer Vision_ , pp. 1849–1856, Sept 2009.  doi: 10.1109/ICCV.2009.5459411.
  * Hoiem et al. (2007) Derek Hoiem, Alexei A. Efros, and Martial Hebert.  Recovering surface layout from an image.  _International Journal of Computer Vision_ , 75(1):151–172, Oct 2007.  ISSN 1573-1405.  doi: 10.1007/s11263-006-0031-y.  URL https://doi.org/10.1007/s11263-006-0031-y.
  * Hutchcroft et al. (2022) Will Hutchcroft, Yuguang Li, Ivaylo Boyadzhiev, Zhiqiang Wan, Haiyan Wang, and Sing Bing Kang.  Covispose: Co-visibility pose transformer for wide-baseline relative pose estimation in 360∘ indoor panoramas.  In _ECCV_ , 2022.
  * Ji et al. (2020) Shunping Ji, Zijie Qin, Jie Shan, and Meng Lu.  Panoramic slam from a multiple fisheye camera rig.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 159:169–183, 2020.  ISSN 0924-2716.  doi: https://doi.org/10.1016/j.isprsjprs.2019.11.014.  URL https://www.sciencedirect.com/science/article/pii/S0924271619302758.
  * Jiang et al. (2021) Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi.  Cotr: Correspondence transformer for matching across images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 6207–6217, October 2021.
  * Jiang et al. (2022) Zhigang Jiang, Zhongzheng Xiang, Jinhua Xu, and Ming Zhao.  Lgt-net: Indoor panoramic room layout estimation with geometry-aware transformer network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1654–1663, June 2022.
  * Lee et al. (2017) Chen-Yu Lee, Vijay Badrinarayanan, Tomasz Malisiewicz, and Andrew Rabinovich.  Roomnet: End-to-end room layout estimation.  _CoRR_ , abs/1703.06241, 2017.  URL http://arxiv.org/abs/1703.06241.
  * Moulon et al. (2016) Pierre Moulon, Pascal Monasse, Romuald Perrot, and Renaud Marlet.  Openmvg: Open multiple view geometry.  In _International Workshop on Reproducible Research in Pattern Recognition_ , pp. 60–74. Springer, 2016.
  * Pagani & Stricker (2011) A. Pagani and D. Stricker.  Structure from motion using full spherical panoramic cameras.  In _2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops)_ , pp. 375–382, 2011.  doi: 10.1109/ICCVW.2011.6130266.
  * Pintore et al. (2018) G. Pintore, F. Ganovelli, R. Pintus, R. Scopigno, and E. Gobbetti.  3d floor plan recovery from overlapping spherical images.  _Computational Visual Media_ , 4:367–383, 2018.
  * Pintore et al. (2019) Giovanni Pintore, Fabio Ganovelli, Alberto Jaspe Villanueva, and Enrico Gobbetti.  Automatic modeling of cluttered multi-room floor plans from panoramic images.  _Computer Graphics Forum_ , 38(7):347–358, 2019\.  doi: https://doi.org/10.1111/cgf.13842.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.13842.
  * Pintore et al. (2020) Giovanni Pintore, Marco Agus, and Enrico Gobbetti.  AtlantaNet: Inferring the 3D indoor layout from a single 360 image beyond the Manhattan world assumption.  In _Proc. ECCV_ , August 2020.  URL http://vic.crs4.it/vic/cgi-bin/bib-page.cgi?id=’Pintore:2020:AI3’.
  * Ramalingam & Brand (2013) Srikumar Ramalingam and Matthew Brand.  Lifting 3d manhattan lines from a single image.  _2013 IEEE International Conference on Computer Vision_ , pp. 497–504, 2013.
  * Salaün et al. (2017) Y. Salaün, R. Marlet, and P. Monasse.  Line-based robust sfm with little image overlap.  In _2017 International Conference on 3D Vision (3DV)_ , pp. 195–204, 2017.  doi: 10.1109/3DV.2017.00031.
  * Schönberger & Frahm (2016) Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Shabani et al. (2021) Mohammad Amin Shabani, Weilian Song, Makoto Odamaki, Hirochika Fujiki, and Yasutaka Furukawa.  Extreme structure from motion for indoor panoramas without visual overlaps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2021.  URL https://aminshabani.github.io/publications/extreme_sfm/pdfs/iccv2021_2088.pdf.
  * Sun et al. (2019) Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen.  Horizonnet: Learning room layout with 1d representation and pano stretch data augmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Taneja et al. (2012) Aparna Taneja, Luca Ballan, and Marc Pollefeys.  Registration of spherical panoramic images with cadastral 3d models.  In _2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission_, pp. 479–486, 2012.  doi: 10.1109/3DIMPVT.2012.45.
  * Wang et al. (2021) Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai.  Led2-net: Monocular 360deg layout estimation via differentiable depth rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 12956–12965, June 2021.
  * Wang et al. (2022) Haiyan Wang, Will Hutchcroft, Yuguang Li, Zhiqiang Wan, Ivaylo Boyadzhiev, Yingli Tian, and Sing Bing Kang.  Psmnet: Position-aware stereo merging network for room layout estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8616–8625, June 2022.
  * Xia et al. (2018) Fei Xia, Amir R. Zamir, Zhi-Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese.  Gibson env: real-world perception for embodied agents.  In _Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on_. IEEE, 2018.
  * Yan et al. (2020) C. Yan, B. Shao, H. Zhao, R. Ning, Y. Zhang, and F. Xu.  3d room layout estimation from a single rgb image.  _IEEE Transactions on Multimedia_ , 22(11):3014–3024, 2020.  doi: 10.1109/TMM.2020.2967645.
  * Yang et al. (2019) Shang-Ta Yang, Fu-En Wang, Chi-Han Peng, Peter Wonka, Min Sun, and Hung-Kuo Chu.  Dula-net: A dual-projection network for estimating room layouts from a single rgb panorama.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Zhang et al. (2014) Yinda Zhang, Shuran Song, Ping Tan, and Jianxiong Xiao.  Panocontext: A whole-room 3d context model for panoramic scene understanding.  In _Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI_ , pp. 668–686, 2014.  doi: 10.1007/978-3-319-10599-4“˙43.  URL https://doi.org/10.1007/978-3-319-10599-4_43.
  * Zheng et al. (2020) Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In _Proceedings of The European Conference on Computer Vision (ECCV)_ , 2020.
  * Zou et al. (2018) Chuhang Zou, Alex Colburn, Qi Shan, and Derek Hoiem.  Layoutnet: Reconstructing the 3d room layout from a single rgb image.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 2051–2059, 2018.
  * Zou et al. (2021) Chuhang Zou, Jheng-Wei Su, Chi-Han Peng, Alex Colburn, Qi Shan, Peter Wonka, Hung-Kuo Chu, and Derek Hoiem.  Manhattan room layout reconstruction from a single 360∘ image: A comparative study of state-of-the-art methods.  _IJCV_ , Feb 2021.  ISSN 1573-1405.  doi: 10.1007/s11263-020-01426-8.  URL https://doi.org/10.1007/s11263-020-01426-8.
  * Özyeşil et al. (2017) Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer.  A survey of structure from motion.  _Acta Numerica_ , 26:305–364, 2017.  doi: 10.1017/S096249291700006X.


* Edgington and Maize (2019) Edgington, S.G., Maize, E.H., 2019\.  Saturn Atmospheric Modeling Working Group Final D-104871.  Technical Report. Jet Propulsion Laboratory.
  * Flasar et al. (2005) Flasar, F.M., Achterberg, R.K., Conrath, B.J., Pearl, J.C., Bjoraker, G.L., Jennings, D.E., Romani, P.N., Simon-Miller, A.A., Kunde, V.G., Nixon, C.A., Bézard, B., Orton, G.S., Spilker, L.J., Spencer, J.R., Irwin, P.G.J., Teanby, N.A., Owen, T.C., Brasunas, J., Segura, M.E., Carlson, R.C., Mamoutkine, A., Gierasch, P.J., Schinder, P.J., Showalter, M.R., Ferrari, C., Barucci, A., Courtin, R., Coustenis, A., Fouchet, T., Gautier, D., Lellouch, E., Marten, A., Prangé, R., Strobel, D.F., Calcutt, S.B., Read, P.L., Taylor, F.W., Bowles, N., Samuelson, R.E., Abbas, M.M., Raulin, F., Ade, P., Edgington, S., Pilorz, S., Wallis, B., Wishnow, E.H., 2005\.  Temperatures, winds, and composition in the saturnian system.  Science 307, 1247–1251.  doi:10.1126/science.1105806.
  * Fletcher et al. (2009) Fletcher, L., Orton, G., Teanby, N., Irwin, P., Bjoraker, G., 2009.  Methane and its isotopologues on saturn from cassini/cirs observations.  Icarus 199, 351–367.  doi:https://doi.org/10.1016/j.icarus.2008.09.019.
  * Fletcher et al. (2018) Fletcher, L.N., Greathouse, T.K., Guerlet, S., Moses, J.I., West, R.A., 2018.  Saturn’s Seasonally Changing Atmosphere: Thermal Structure, Composition and Aerosols. Cambridge University Press.  Cambridge Planetary Science, p. 251–294.  doi:10.1017/9781316227220.010.
  * Fletcher et al. (2015) Fletcher, L.N., Irwin, P., Sinclair, J., Orton, G., Giles, R., Hurley, J., Gorius, N., Achterberg, R., Hesman, B., Bjoraker, G., 2015\.  Seasonal evolution of saturn’s polar temperatures and composition.  Icarus 250, 131–153.  doi:https://doi.org/10.1016/j.icarus.2014.11.022.
  * Fletcher et al. (2023) Fletcher, L.N., King, O.R.T., Harkett, J., Hammel, H.B., Roman, M.T., Melin, H., Hedman, M.M., Moses, J.I., Guerlet, S., Milam, S.N., Tiscareno, M.S., 2023.  Saturn’s Atmosphere in Northern Summer Revealed by JWST/MIRI.  Journal of Geophysical Research: Planets 128, e2023JE007924.  doi:10.1029/2023JE007924.
  * Fletcher et al. (2020) Fletcher, L.N., Sromovsky, L., Hue, V., Moses, J.I., Guerlet, S., West, R.A., Koskinen, T., 2020.  Saturn’s Seasonal Atmosphere at Northern Summer Solstice.  doi:10.48550/ARXIV.2012.09288. publisher: arXiv Version Number: 3.
  * Florescumitchell and Mitchell (2006) Florescumitchell, A., Mitchell, J., 2006\.  Dissociative recombination.  Physics Reports 430, 277–374.  doi:10.1016/j.physrep.2006.04.002.
  * Friedson (2002) Friedson, A., 2002.  Models for Polar Haze Formation in Jupiter’s Stratosphere.  Icarus 158, 389–400.  doi:10.1006/icar.2002.6885.
  * Friedson and Moses (2012) Friedson, A.J., Moses, J.I., 2012\.  General circulation and transport in Saturn’s upper troposphere and stratosphere.  Icarus 218, 861–875.  doi:10.1016/j.icarus.2012.02.004.
  * Fuller-Rowell (1998) Fuller-Rowell, T.J., 1998.  The “thermospheric spoon”: A mechanism for the semiannual density variation.  Journal of Geophysical Research: Space Physics 103, 3951–3956.  doi:10.1029/97JA03335.
  * Galand et al. (2011) Galand, M., Moore, L., Mueller-Wodarg, I., Mendillo, M., Miller, S., 2011.  Response of Saturn’s auroral ionosphere to electron precipitation: Electron density, electron temperature, and electrical conductivity: SATURN’S AURORAL IONOSPHERE.  Journal of Geophysical Research: Space Physics 116, n/a–n/a.  doi:10.1029/2010JA016412.
  * Greathouse et al. (2005) Greathouse, T.K., Lacy, J.H., Bézard, B., Moses, J.I., Griffith, C.A., Richter, M.J., 2005\.  Meridional variations of temperature, C2H2 and C2H6 abundances in Saturn’s stratosphere at southern summer solstice.  Icarus 177, 18–31.  doi:10.1016/j.icarus.2005.02.016.
  * Guerlet et al. (2011) Guerlet, S., Fouchet, T., Bézard, B., Flasar, F.M., Simon-Miller, A.A., 2011.  Evolution of the equatorial oscillation in saturn’s stratosphere between 2005 and 2010 from cassini/cirs limb data analysis.  Geophysical Research Letters 38.  doi:https://doi.org/10.1029/2011GL047192.
  * Guerlet et al. (2010) Guerlet, S., Fouchet, T., Bézard, B., Moses, J.I., Fletcher, L.N., Simon-Miller, A.A., Michael Flasar, F., 2010.  Meridional distribution of CH3C2H and C4H2 in Saturn’s stratosphere from CIRS/Cassini limb and nadir observations.  Icarus 209, 682–695.  doi:10.1016/j.icarus.2010.03.033.
  * Guerlet et al. (2009) Guerlet, S., Fouchet, T., Bézard, B., Simon-Miller, A.A., Michael Flasar, F., 2009.  Vertical and meridional distribution of ethane, acetylene and propane in saturn’s stratosphere from cirs/cassini limb observations.  Icarus 203, 214–232.  doi:https://doi.org/10.1016/j.icarus.2009.04.002.
  * Guerlet et al. (2018) Guerlet, S., Fouchet, T., Spiga, A., Flasar, F.M., Fletcher, L.N., Hesman, B.E., Gorius, N., 2018.  Equatorial oscillation and planetary wave activity in saturn’s stratosphere through the cassini epoch.  Journal of Geophysical Research: Planets 123, 246–261.  doi:https://doi.org/10.1002/2017JE005419.
  * Guerlet et al. (2015) Guerlet, S., Fouchet, T., Vinatier, S., Simon, A.A., Dartois, E., Spiga, A., 2015\.  Stratospheric benzene and hydrocarbon aerosols detected in Saturn’s auroral regions.  Astronomy & Astrophysics 580, A89.  doi:10.1051/0004-6361/201424745.
  * Guerlet et al. (2014) Guerlet, S., Spiga, A., Sylvestre, M., Indurain, M., Fouchet, T., Leconte, J., Millour, E., Wordsworth, R., Capderou, M., Bézard, B., Forget, F., 2014.  Global climate modeling of Saturn’s atmosphere. Part I: Evaluation of the radiative transfer model.  Icarus 238, 110–124.  doi:10.1016/j.icarus.2014.05.010.
  * Gérard et al. (2009) Gérard, J.C., Bonfond, B., Gustin, J., Grodent, D., Clarke, J.T., Bisikalo, D., Shematovich, V., 2009.  Altitude of Saturn’s aurora and its implications for the characteristic energy of precipitated electrons: ALTITUDE OF SATURN’S AURORA.  Geophysical Research Letters 36, n/a–n/a.  doi:10.1029/2008GL036554.
  * Hanel et al. (1981) Hanel, R., Conrath, B., Flasar, F.M., Kunde, V., Maguire, W., Pearl, J., Pirraglia, J., Samuelson, R., Herath, L., Allison, M., Cruikshank, D., Gautier, D., Gierasch, P., Horn, L., Koppany, R., Ponnamperuma, C., 1981\.  Infrared Observations of the Saturnian System from Voyager 1.  Science 212, 192–200.  doi:10.1126/science.212.4491.192.
  * Hanel et al. (1983) Hanel, R., Conrath, B., Kunde, V., Pearl, J., Pirraglia, J., 1983.  Albedo, internal heat flux, and energy balance of Saturn.  Icarus 53, 262–285.  doi:10.1016/0019-1035(83)90147-1.
  * Hesman et al. (2009) Hesman, B.E., Jennings, D.E., Sada, P.V., Bjoraker, G.L., Achterberg, R.K., Simon-Miller, A.A., Anderson, C.M., Boyle, R.J., Nixon, C.A., Fletcher, L.N., McCabe, G.H., 2009.  Saturn’s latitudinal C2H2 and C2H6 abundance profiles from Cassini/CIRS and ground-based observations.  Icarus 202, 249–259.  doi:10.1016/j.icarus.2009.02.013.
  * Holberg et al. (1982) Holberg, J.B., Forrester, W.T., Lissauer, J.J., 1982.  Identification of resonance features within the rings of Saturn.  Nature 297, 115–120.  doi:10.1038/297115a0.
  * Howett et al. (2007) Howett, C., Irwin, P., Teanby, N., Simonmiller, A., Calcutt, S., Fletcher, L., Dekok, R., 2007.  Meridional variations in stratospheric acetylene and ethane in the southern hemisphere of the saturnian atmosphere as determined from Cassini/CIRS measurements.  Icarus 190, 556–572.  doi:10.1016/j.icarus.2007.03.009.
  * Hue et al. (2015) Hue, V., Cavalié, T., Dobrijevic, M., Hersant, F., Greathouse, T., 2015.  2D photochemical modeling of Saturn’s stratosphere. Part I: Seasonal variation of atmospheric composition without meridional transport.  Icarus 257, 163–184.  doi:10.1016/j.icarus.2015.04.001.
  * Hue et al. (2016) Hue, V., Greathouse, T., Cavalié, T., Dobrijevic, M., Hersant, F., 2016.  2D photochemical modeling of Saturn’s stratosphere. Part II: Feedback between composition and temperature.  Icarus 267, 334–343.  doi:10.1016/j.icarus.2015.12.007.
  * Kim et al. (2017) Kim, S.J., Geballe, T.R., Greathouse, T.K., Yung, Y.L., Miller, S., Orton, G., Minh, Y., 2017.  Temperatures and CH4 mixing ratios near the homopause of the 8 µm north polar hot spot of Jupiter.  Icarus 281, 281–285.  doi:10.1016/j.icarus.2016.09.017.
  * Kim et al. (2014) Kim, S.J., Sim, C., Sohn, M., Moses, J., 2014.  CH4 mixing ratios at microbar pressure levels of Jupiter as constrained by 3-micron ISO data.  Icarus 237, 42–51.  doi:10.1016/j.icarus.2014.04.023.
  * Koskinen and Guerlet (2018) Koskinen, T., Guerlet, S., 2018\.  Atmospheric structure and helium abundance on saturn from cassini/uvis and cirs observations.  Icarus 307, 161–171.  doi:10.1016/j.icarus.2018.02.020.
  * Koskinen et al. (2013) Koskinen, T., Sandel, B., Yelle, R., Capalbo, F., Holsclaw, G., McClintock, W., Edgington, S., 2013.  The density and temperature structure near the exobase of saturn from cassini uvis solar occultations.  Icarus 226, 1318–1330.  doi:https://doi.org/10.1016/j.icarus.2013.07.037.
  * Koskinen et al. (2020) Koskinen, T., Sandel, B., Yelle, R., Holsclaw, G., Quemerais, E., 2020.  Saturn in lyman alpha: A comparison of Cassini and Voyager observations.  Icarus 339, 113594\.  doi:10.1016/j.icarus.2019.113594.
  * Koskinen et al. (2015) Koskinen, T., Sandel, B., Yelle, R., Strobel, D., Müller-Wodarg, I., Erwin, J., 2015\.  Saturn’s variable thermosphere from Cassini/UVIS occultations.  Icarus 260, 174–189.  doi:10.1016/j.icarus.2015.07.008.
  * Koskinen et al. (2011) Koskinen, T., Yelle, R., Snowden, D., Lavvas, P., Sandel, B., Capalbo, F., Benilan, Y., West, R., 2011\.  The mesosphere and lower thermosphere of titan revealed by cassini/uvis stellar occultations.  Icarus 216, 507–534.  doi:https://doi.org/10.1016/j.icarus.2011.09.022.
  * Koskinen et al. (2016) Koskinen, T.T., Moses, J.I., West, R.A., Guerlet, S., Jouchoux, A., 2016.  The detection of benzene in saturn’s upper atmosphere.  Geophysical Research Letters 43, 7895–7901.  doi:https://doi.org/10.1002/2016GL070000.
  * Koskinen et al. (2021) Koskinen, T.T., Strobel, D.F., Brown, Z., 2021.  An empirical model of the Saturn thermosphere.  Icarus 362, 114396\.  doi:10.1016/j.icarus.2021.114396.
  * Lamy et al. (2018) Lamy, L., Prangé, R., Tao, C., Kim, T., Badman, S.V., Zarka, P., Cecconi, B., Kurth, W.S., Pryor, W., Bunce, E., Radioti, A., 2018.  Saturn northern aurorae at solstice from HST observations coordinated with Cassini Grand Finale.  Geophysical Research Letters 45, 9353–9362.  doi:10.1029/2018GL078211. arXiv:1808.08168 [astro-ph].
  * Larsen and Meriwether (2012) Larsen, M.F., Meriwether, J.W., 2012\.  Vertical winds in the thermosphere.  Journal of Geophysical Research 117\.
  * Lindal et al. (1985) Lindal, G.F., Sweetnam, D.N., Eshleman, V.R., 1985.  The atmosphere of Saturn - an analysis of the Voyager radio occultation measurements.  The Astronomical Journal 90, 1136\.  doi:10.1086/113820.
  * Lindzen (1981) Lindzen, R.S., 1981.  Turbulence and stress owing to gravity wave and tidal breakdown.  Journal of Geophysical Research 86, 9707.  doi:10.1029/JC086iC10p09707.
  * Loison et al. (2015) Loison, J., Hébrard, E., Dobrijevic, M., Hickson, K., Caralp, F., Hue, V., Gronoff, G., Venot, O., Bénilan, Y., 2015.  The neutral photochemistry of nitriles, amines and imines in the atmosphere of Titan.  Icarus 247, 218–247.  doi:10.1016/j.icarus.2014.09.039.
  * Loison et al. (2017) Loison, J.C., Agúndez, M., Wakelam, V., Roueff, E., Gratier, P., Marcelino, N., Reyes, D.N., Cernicharo, J., Gerin, M., 2017.  The interstellar chemistry of C3H and C3H2 isomers.  Monthly Notices of the Royal Astronomical Society 470, 4075–4088.  doi:10.1093/mnras/stx1265.
  * Loison et al. (2019) Loison, J.C., Dobrijevic, M., Hickson, K.M., 2019.  The photochemical production of aromatics in the atmosphere of Titan.  icarus 329, 55–71.  doi:10.1016/j.icarus.2019.03.024.
  * Markwardt (2009) Markwardt, C.B., 2009.  Non-linear Least Squares Fitting in IDL with MPFIT.
  * Marrero and Mason (1972) Marrero, T.R., Mason, E.A., 1972\.  Gaseous diffusion coefficients.  Journal of Physical and Chemical Reference Data 1, 3–118.  doi:10.1063/1.3253094, arXiv:https://doi.org/10.1063/1.3253094.
  * McEwan and Anicich (2007) McEwan, M.J., Anicich, V.G., 2007\.  Titan’s ion chemistry: A laboratory perspective.  Mass Spectrometry Reviews 26, 281–319.  doi:10.1002/mas.20117.
  * Medvedev and Yiğit (2019) Medvedev, A.S., Yiğit, E., 2019\.  Gravity Waves in Planetary Atmospheres: Their Effects and Parameterization in Global Circulation Models.  Atmosphere 10, 531\.  doi:10.3390/atmos10090531.
  * Militzer et al. (2019) Militzer, B., Wahl, S., Hubbard, W.B., 2019.  Models of Saturn’s Interior Constructed with Accelerated Concentric Maclaurin Spheroid Method.  The Astrophysical Journal 879, 78\.  doi:10.3847/1538-4357/ab23f0. arXiv:1905.08907 [astro-ph].
  * Moos and Clarke (1979) Moos, H.W., Clarke, J.T., 1979\.  Detection of acetylene in the Saturnian atmosphere, using the IUE satellite.  The Astrophysical Journal 229, L107.  doi:10.1086/182939.
  * Moses (2000) Moses, J., 2000.  Photochemistry of Saturn’s Atmosphere II. Effects of an Influx of External Oxygen.  Icarus 145, 166–202.  doi:10.1006/icar.1999.6320.
  * Moses et al. (2015) Moses, J.I., Armstrong, E.S., Fletcher, L.N., Friedson, A.J., Irwin, P.G., Sinclair, J.A., Hesman, B.E., 2015.  Evolution of stratospheric chemistry in the Saturn storm beacon region.  Icarus 261, 149–168.  doi:10.1016/j.icarus.2015.08.012.
  * Moses et al. (2023) Moses, J.I., Brown, Z.L., Koskinen, T.T., Fletcher, L.N., Serigano, J., Guerlet, S., Moore, L., Waite, J.H., Ben-Jaffel, L., Galand, M., Chadney, J.M., Hörst, S.M., Sinclair, J.A., Vuitton, V., Müller-Wodarg, I., 2023.  Saturn’s atmospheric response to the large influx of ring material inferred from cassini inms measurements.  Icarus 391, 115328\.  doi:https://doi.org/10.1016/j.icarus.2022.115328.
  * Moses et al. (2018) Moses, J.I., Fletcher, L.N., Greathouse, T.K., Orton, G.S., Hue, V., 2018.  Seasonal stratospheric photochemistry on Uranus and Neptune.  Icarus 307, 124–145.  doi:10.1016/j.icarus.2018.02.004.
  * Moses et al. (2005) Moses, J.I., Fouchet, T., Bézard, B., Gladstone, G.R., Lellouch, E., Feuchtgruber, H., 2005\.  Photochemistry and diffusion in Jupiter’s stratosphere: Constraints from ISO observations and comparisons with other giant planets: PHOTOCHEMISTRY IN JUPITER’S STRATOSPHERE.  Journal of Geophysical Research: Planets 110, n/a–n/a.  doi:10.1029/2005JE002411.
  * Moses and Greathouse (2005) Moses, J.I., Greathouse, T.K., 2005\.  Latitudinal and seasonal models of stratospheric photochemistry on saturn: Comparison with infrared data from irtf/texes.  Journal of Geophysical Research: Planets 110\.  doi:https://doi.org/10.1029/2005JE002450.
  * Moses et al. (2013) Moses, J.I., Madhusudhan, N., Visscher, C., Freedman, R.S., 2013\.  CHEMICAL CONSEQUENCES OF THE C/O RATIO ON HOT JUPITERS: EXAMPLES FROM WASP-12b, CoRoT-2b, XO-1b, AND HD 189733b.  The Astrophysical Journal 763, 25\.  doi:10.1088/0004-637X/763/1/25.
  * Moses and Poppe (2017) Moses, J.I., Poppe, A.R., 2017\.  Dust ablation on the giant planets: Consequences for stratospheric photochemistry.  Icarus 297, 33–58.  doi:10.1016/j.icarus.2017.06.002.
  * Moses et al. (2011) Moses, J.I., Visscher, C., Fortney, J.J., Showman, A.P., Lewis, N.K., Griffith, C.A., Klippenstein, S.J., Shabram, M., Friedson, A.J., Marley, M.S., Freedman, R.S., 2011.  DISEQUILIBRIUM CARBON, OXYGEN, AND NITROGEN CHEMISTRY IN THE ATMOSPHERES OF HD 189733b AND HD 209458b.  The Astrophysical Journal 737, 15\.  doi:10.1088/0004-637X/737/1/15.
  * Müller-Wodarg et al. (2006) Müller-Wodarg, I., Mendillo, M., Yelle, R., Aylward, A., 2006\.  A global circulation model of Saturn’s thermosphere.  Icarus 180, 147–160.  doi:10.1016/j.icarus.2005.09.002.
  * Müller-Wodarg et al. (2012) Müller-Wodarg, I., Moore, L., Galand, M., Miller, S., Mendillo, M., 2012.  Magnetosphere–atmosphere coupling at Saturn: 1 – Response of thermosphere and ionosphere to steady state polar forcing.  Icarus 221, 481–494.  doi:10.1016/j.icarus.2012.08.034.
  * Müller-Wodarg (2003) Müller-Wodarg, I.C.F., 2003.  On the global distribution of neutral gases in Titan’s upper atmosphere and its effect on the thermal structure.  Journal of Geophysical Research 108, 1453.  doi:10.1029/2003JA010054.
  * Müller-Wodarg et al. (2019) Müller-Wodarg, I.C.F., Koskinen, T.T., Moore, L., Serigano, J., Yelle, R.V., Hörst, S., Waite, J.H., Mendillo, M., 2019\.  Atmospheric Waves and Their Possible Effect on the Thermal Structure of Saturn’s Thermosphere.  Geophysical Research Letters 46, 2372–2380.  doi:10.1029/2018GL081124.
  * Reid et al. (1987) Reid, R.C., Prausnitz, J.M., Poling, B.E., 1987.  The properties of gases and liquids.  4th ed ed., McGraw-Hill, New York.
  * Sandel et al. (1982) Sandel, B.R., Shemansky, D.E., Broadfoot, A.L., Holberg, J.B., Smith, G.R., McConnell, J.C., Strobel, D.F., Atreya, S.K., Donahue, T.M., Moos, H.W., Hunten, D.M., Pomphrey, R.B., Linick, S., 1982.  Extreme Ultraviolet Observations from the Voyager 2 Encounter with Saturn.  Science 215, 548–553.  doi:10.1126/science.215.4532.548.
  * Schunk and Nagy (2000) Schunk, R.W., Nagy, A.F., 2000\.  Ionospheres: Physics, Plasma Physics, and Chemistry.  1 ed., Cambridge University Press.  doi:10.1017/CBO9780511551772.
  * Seiff et al. (1998) Seiff, A., Kirk, D.B., Knight, T.C.D., Young, R.E., Mihalov, J.D., Young, L.A., Milos, F.S., Schubert, G., Blanchard, R.C., Atkinson, D., 1998\.  Thermal structure of Jupiter’s atmosphere near the edge of a 5-micron hot spot in the north equatorial belt.  Journal of Geophysical Research: Planets 103, 22857–22889.  doi:10.1029/98JE01766.
  * Shemansky and Liu (2012) Shemansky, D., Liu, X., 2012\.  Saturn upper atmospheric structure from Cassini EUV and FUV occultations ${}^{\textrm{1}}$ This article is part of a Special Issue that honours the work of Dr. Donald M. Hunten FRSC who passed away in December 2010 after a very illustrious career.  Canadian Journal of Physics 90, 817–831.  doi:10.1139/p2012-036.
  * Shemansky et al. (2009) Shemansky, D., Liu, X., Melin, H., 2009.  The Saturn hydrogen plume.  Planetary and Space Science 57, 1659–1670.  doi:10.1016/j.pss.2009.05.002.
  * Sinclair et al. (2014) Sinclair, J., Irwin, P., Fletcher, L., Greathouse, T., Guerlet, S., Hurley, J., Merlet, C., 2014.  From Voyager-IRIS to Cassini-CIRS: Interannual variability in Saturn’s stratosphere?  Icarus 233, 281–292.  doi:10.1016/j.icarus.2014.02.009.
  * Sinclair et al. (2013) Sinclair, J., Irwin, P., Fletcher, L., Moses, J., Greathouse, T., Friedson, A., Hesman, B., Hurley, J., Merlet, C., 2013.  Seasonal variations of temperature, acetylene and ethane in Saturn’s atmosphere from 2005 to 2010, as observed by Cassini-CIRS.  Icarus 225, 257–271.  doi:10.1016/j.icarus.2013.03.011.
  * Sinclair et al. (2017) Sinclair, J., Orton, G., Greathouse, T., Fletcher, L., Moses, J., Hue, V., Irwin, P., 2017.  Jupiter’s auroral-related stratospheric heating and chemistry I: Analysis of Voyager-IRIS and Cassini-CIRS spectra.  Icarus 292, 182–207.  doi:10.1016/j.icarus.2016.12.033.
  * Sinclair et al. (2020) Sinclair, J.A., Greathouse, T.K., Giles, R.S., Antuñano, A., Moses, J.I., Fouchet, T., Bézard, B., Tao, C., Martín-Torres, J., Clark, G.B., Grodent, D., Orton, G.S., Hue, V., Fletcher, L.N., Irwin, P.G.J., 2020.  Spatial Variations in the Altitude of the CH ${}_{\textrm{4}}$ Homopause at Jupiter’s Mid-to-high Latitudes, as Constrained from IRTF-TEXES Spectra.  The Planetary Science Journal 1, 85\.  doi:10.3847/PSJ/abc887.
  * Stevens et al. (1993) Stevens, M.H., Strobel, D.F., Herbert, F., 1993.  An Analysis of the Voyager 2 Ultraviolet Spectrometer Occultation Data at Uranus: Inferring Heat Sources and Model Atmospheres.  Icarus 101, 45–63.  doi:10.1006/icar.1993.1005.
  * Strobel et al. (2018) Strobel, D.F., Koskinen, T.T., Müller-Wodarg, I., 2018.  Saturn’s Variable Thermosphere. 1 ed.. Cambridge University Press.  pp. 224–250.  doi:10.1017/9781316227220.009.
  * Swenson et al. (2019) Swenson, G.R., Salinas, C.C.J.H., Vargas, F., Zhu, Y., Kaufmann, M., Jones, M., Drob, D.P., Liu, A., Yue, J., Yee, J.H., 2019.  Determination of Global Mean Eddy Diffusive Transport in the Mesosphere and Lower Thermosphere From Atomic Oxygen and Carbon Dioxide Climatologies.  Journal of Geophysical Research: Atmospheres 124, 13519–13533.  doi:10.1029/2019JD031329.
  * Sylvestre et al. (2015) Sylvestre, M., Guerlet, S., Fouchet, T., Spiga, A., Flasar, F., Hesman, B., Bjoraker, G., 2015.  Seasonal changes in saturn’s stratosphere inferred from cassini/cirs limb observations.  Icarus 258, 224–238.  doi:https://doi.org/10.1016/j.icarus.2015.05.025.
  * Tikhonov and Arsenin (1977) Tikhonov, A.N., Arsenin, V.Y., 1977\.  Solutions of ill-posed problems.  V. H. Winston & Sons, Washington, D.C.: John Wiley & Sons, New York.  Translated from the Russian, Preface by translation editor Fritz John, Scripta Series in Mathematics.
  * Tokunaga et al. (1975) Tokunaga, A., Knacke, R.F., Owen, T., 1975.  The detection of ethane on Saturn.  The Astrophysical Journal 197, L77.  doi:10.1086/181782.
  * Trafton (1973) Trafton, L., 1973.  Saturn: a Study of the 3V3 Methane Band.  The Astrophysical Journal 182, 615\.  doi:10.1086/152170.
  * Trafton (1977) Trafton, L., 1977.  Saturn: Long-term variation of H2 and CH4 absorptions.  Icarus 31, 369–384.  doi:10.1016/0019-1035(77)90029-X.
  * Vervack and Moses (2015) Vervack, R.J., Moses, J.I., 2015\.  Saturn’s upper atmosphere during the Voyager era: Reanalysis and modeling of the UVS occultations.  Icarus 258, 135–163.  doi:10.1016/j.icarus.2015.06.007.
  * Vuitton et al. (2019) Vuitton, V., Yelle, R., Klippenstein, S., Hörst, S., Lavvas, P., 2019.  Simulating the density of organic species in the atmosphere of Titan with a coupled ion-neutral photochemical model.  Icarus 324, 120–197.  doi:10.1016/j.icarus.2018.06.013.
  * Vuitton et al. (2008) Vuitton, V., Yelle, R.V., Cui, J., 2008.  Formation and distribution of benzene on titan.  Journal of Geophysical Research 113, E05007.  doi:10.1029/2007JE002997.
  * Vuitton et al. (2012) Vuitton, V., Yelle, R.V., Lavvas, P., Klippenstein, S.J., 2012\.  RAPID ASSOCIATION REACTIONS AT LOW PRESSURE: IMPACT ON THE FORMATION OF HYDROCARBONS ON TITAN.  The Astrophysical Journal 744, 11\.  doi:10.1088/0004-637X/744/1/11.
  * Waite et al. (2018) Waite, J.H., Perryman, R.S., Perry, M.E., Miller, K.E., Bell, J., Cravens, T.E., Glein, C.R., Grimes, J., Hedman, M., Cuzzi, J., Brockwell, T., Teolis, B., Moore, L., Mitchell, D.G., Persoon, A., Kurth, W.S., Wahlund, J.E., Morooka, M., Hadid, L.Z., Chocron, S., Walker, J., Nagy, A., Yelle, R., Ledvina, S., Johnson, R., Tseng, W., Tucker, O.J., Ip, W.H., 2018\.  Chemical interactions between Saturn’s atmosphere and its rings.  Science 362, eaat2382.  doi:10.1126/science.aat2382.
  * Wong (2002) Wong, A.S., 2002.  Photochemical studies of Jupiter and Titan.  Ph.D. thesis. California Institute of Technology.  Pages: 1397 Publication Title: Ph.D. Thesis ADS Bibcode: 2002PhDT……..21W.
  * Wong et al. (2003) Wong, A.S., Yung, Y.L., Friedson, A.J., 2003.  Benzene and Haze Formation in the Polar Atmosphere of Jupiter: BENZENE AND HAZE FORMATION.  Geophysical Research Letters 30.  doi:10.1029/2002GL016661.
  * Wu et al. (2001) Wu, C.Y.R., Chen, F.Z., Judge, D.L., 2001.  Measurements of temperature-dependent absorption cross sections of C ${}_{\textrm{2}}$ H ${}_{\textrm{2}}$ in the VUV-UV region.  Journal of Geophysical Research: Planets 106, 7629–7636.  doi:10.1029/2000JE001292.
  * Wu et al. (2004) Wu, C.Y.R., Chen, F.Z., Judge, D.L., 2004.  Temperature-dependent photoabsorption cross sections in the vuv-uv region: Ethylene.  Journal of Geophysical Research: Planets 109\.  doi:https://doi.org/10.1029/2003JE002180.
  * Yelle et al. (2001) Yelle, R., Griffith, C.A., Young, L.A., 2001.  Structure of the Jovian Stratosphere at the Galileo Probe Entry Site.  Icarus 152, 331–346.  doi:10.1006/icar.2001.6640.
  * Yelle et al. (2008) Yelle, R.V., Cui, J., Müller-Wodarg, I.C.F., 2008.  Methane escape from Titan’s atmosphere.  Journal of Geophysical Research (Planets) 113, E10003.  doi:10.1029/2007JE003031.
  * Yelle et al. (1993) Yelle, R.V., Herbert, F., Sandel, B.R., Vervack, R.J., Wentzel, T.M., 1993.  The Distribution Hydrocarbons in Neptune’s Upper Atmosphere.  Icarus 104, 38–59.  doi:10.1006/icar.1993.1081.
  * Yung et al. (1984) Yung, Y., Allen, M., Pinto, J.P., 1984.  Photochemistry of the atmosphere of titan - comparison between model and obserations.  Astrophysical Journal Supplement Series 55, 465–506.  doi:doi:10.1086/190963.
  * Zhang and Showman (2018) Zhang, X., Showman, A.P., 2018\.  Global-mean Vertical Tracer Mixing in Planetary Atmospheres. I. Theory and Fast-rotating Planets.  The Astrophysical Journal 866, 1\.  doi:10.3847/1538-4357/aada85.


=

# Occupancy Grid Based Reactive Planner

1st Benjamin Hall  Robotics & Mechatronics
Marietta, United States
<EMAIL_ADDRESS>2nd Andrew Goeden  Computer Science
Marietta, United States
<EMAIL_ADDRESS>3rd Sahan Reddy  Computer Science
Marietta, United States
<EMAIL_ADDRESS>4th Timothy Gallion  Electrical Engineering
Marietta, United States
<EMAIL_ADDRESS>5th Charles Koduru  Robotics & Mechatronics
Marietta, USA
<EMAIL_ADDRESS>6th M. Hassan Tanveer  Robotics & Mechatronics
Marietta, USA
<EMAIL_ADDRESS>
###### Abstract

This paper proposes a perception and path planning pipeline for autonomous
racing in an unknown bounded course. The pipeline was initially created for
the 2021 evGrandPrix autonomous division and was further improved for the 2022
event, both of which resulting in first place finishes. Using a simple LiDAR-
based perception pipeline feeding into an occupancy grid based expansion
algorithm, we determine a goal point to drive. This pipeline successfully
achieved reliable and consistent laps in addition with occupancy grid
algorithm to know the ways around a cone-defined track with an averaging
speeds of 6.85 m/s over a distance 434.2 meters for a total lap time of 63.4
seconds.

###### Index Terms:

Autonomous Driving, Occupancy Grid, Reactive Planner, Ground Vehicle,
Navigation

##  I Introduction

Since vehicles have existed, they have been raced against one another. Racing
has been the breeding ground for many innovations in the automotive industry
that we take for granted today, such as more efficient engines, lighter and
stronger materials, and safety elements[1]. Autonomous racing seeks to
replicate this success of traditional racing in contributing to driven
vehicles but instead to the field of autonomy[2]. Autonomous racing,
generally, can be defined as either a head-to-head or fastest solo lap around
a track by a vehicle that is controlling itself[3].

Figure 1: EVT Autonomous Go-Kart Driving On Track

Our competition is based on how fast an autonomous go-kart can go around a
cone-defined track. The kart cannot start with a map of the track, but mapping
can be done throughout the race[4]. This paper focuses on the reactive planner
that is used during the first lap of the race before we have mapped out the
track. The approaches to solve the challenges in this competition, such as the
first lap track exploration problem, were inspired by solutions from other
competitions such as F1TENTH and Formula Student Driverless.

The Formula Student Driverless competition contains the similar problem of
navigating an unknown track. Within Formula Student Driverless, the track is
defined by two sides of different colored cones, each color representing a
different side of the track. Within F1tenth, the track is defined by
continuous walls. F1tenth has solved the problem of navigation around the
track with approaches such as follow-the-gap [5]. Within Formula Student
Driverless, solutions most often include cone detection and classification and
since the inside and outside of the track are known at all points along the
track the solution for determining a path to navigate within becomes trivial
[6]. Within evGrandPrix, the track is defined by semi-continuous walls of the
same color cone[4].

Compared to many Formula Student Driverless solutions to the problem, our
approach does not require object classification or a pre-defined inside and
outside track [6]. We also do not generate many different possibles paths each
time like a Monte-Carlo approach, but instead create a single path each time
[7]. Similar to follow-the-gap our planning stage requires a continuous track
edge definition, however our perception stage produces a simulated false wall.
Our planning and perception stages are loosely coupled, with the only
constraint being that the perception stage must output a continuous wall on at
least one side of the track at all times.

##  II Autonomous Driving Pipeline

###  II-A Overview

Our proposed autonomous driving pipeline uses a local perception, planning and
control strategy to drive around a bounded track. In our context, local refers
to the coordinate system with its origin located on the kart itself. The
perception pipeline is expected to create an occupancy grid with continuous
boundaries on the left and right of the kart. Then, the path planner selects a
goal point on this occupancy grid, which is driven to with our pure pursuit
controller. The pipeline was implemented leveraging ROS 2 Galactic [8] along
with the Point Cloud Library[9], OpenCV [10] and Eigen [11].

Figure 2: Pipeline Overview

###  II-B Perception

In the evGrandPrix competition, the track is defined by traffic cones spaced
approximately 2 meters apart. To perceive this track boundary, we take an
entirely LiDAR-based approach with a Velodyne VLP-16 [12]. First, the ground
is segmented out of the pointcloud using RANSAC [13]—we assume that any points
left over are obstacles that define our boundary. These non-ground points are
flattened into an occupancy grid. Finally, this occupancy grid is inflated
with a Gaussian blur in order to (1) create a continuous wall from the
discrete cones and (2) create a ”cost valley” in the center of the track so
that the planner follows a rough centerline. The result of this can be seen in
Fig. 3.

Figure 3: Blurred Occupancy Grid

###  II-C Path Planning

(a) Initial

(b) Expansion space

(c) Identify cells with the least cost

(d) Select the one with the least deviation

(e) Repeat until maximum expansions is reached

Figure 4: The pipeline scans each row and identifies the spaces with least
cost and then selects the space of least cost with the least deviation before
repeating on the next row.

Once we have created an occupancy grid that has continuous walls, we can start
to plan our path. Our planning algorithm (Algorithm 1) works by completing a
set number of expansions starting from an initial node. This node is at the
location of the LiDAR, which is the center of the grid (Fig. 4(a)).

The structure of each expansion follows suit: we first define the subspace
(row) ahead of the LiDAR in which we plan to scan for least cost (Fig. 4(b)).
This subspace is defined by $check\\_width$ which is centered from the
previous node (Fig. 4(c)). After finding the points with the least costs, if
there are multiple with the same cost then we select the point with the least
deviation from the previous/current node (Fig. 4(d)) to be the next node in
the path, otherwise we select the singular point of least cost as the next
node (Fig. 4(e)).

The path creation is done via a point-by-point expansion from the center of
the grid which is the location of the go-kart. The zone we are allowed to
expand to is above and an amount to the left and right of this initial point.
Then the cell inside of this zone that we expand to is chosen in two steps:
first, whichever one has least cost and second if there is a tie between
multiple cells, the cell closest to the center of the expansion zone, and
therefore the cell we’re expanding from, is chosen.

This expansion process repeats a set number of times, $n$. The last point in
this series of expansions is chosen as our goal point to drive to. This
process can be seen in figure 4. The algorithm that describes this can be seen
in Algorithm 1.

We do this using pure pursuit [14], which controls our steering angle while
driving forward at a constant velocity.

Algorithm 1 Path Planning Algorithm

Occupancy Grid

Goal Point

$x\leftarrow grid\\_width/2$, current node x

$y\leftarrow grid\\_height/2$, current node y

$half\\_width$, half the configured width of the expansion space

for $n\leftarrow 0$ to $max\\_expansions$ do

$x^{\prime}$

$y^{\prime}\leftarrow y+1$

$least\\_cost\leftarrow\infty$

$least\\_x\\_deviation\leftarrow\infty$

for $c\leftarrow-half\\_width$ to $half\\_width$ do

$expand\\_x$ $\leftarrow$ $c+x$

if $grid[y^{\prime}][expand\\_x]\leq least\\_cost$ and
$|x-expand\\_x|<least\\_x\\_deviation$ then

$least\\_cost\leftarrow grid[y^{\prime}][expand\\_x]$

$least\\_x\\_deviation$$\leftarrow|x-expand\\_x|$

$x^{\prime}\leftarrow expand\\_x$

end if

end for

$x\leftarrow x^{\prime}$

$y\leftarrow y^{\prime}$

end for

##  III Results and Discussion

Using this pipeline, the kart was able to fully navigate the track with
surprising consistency: over the five laps of the race, the completion time
variance was within a quarter second. For these laps, our average speed was
6.85 m/s; there is much room for improvement here, since our pure pursuit goal
point follower runs at a constant speed.

Because the planner has to move forward in the occupancy grid with every
expansion and it has a limited space that it can expand into, in a case where
the only options in the expansion space are further into a high cost boundary,
it will choose a goal point past that boundary. This means that it will fail
when presented with a wall perfectly perpendicular to the kart or one in an
upside down ”U” shape in front of the kart. For our use case, we can assume
that this happens extremely rarely, since a track has smooth curves.

One possible improvement to this pipeline is addition that could tune the
occupancy grid blur dynamically to always produce an optimal ”cost valley” so
that the planner plans along a rough center line. This would greatly reduce
manual tuning time and would improve performance on width-varying tracks. In
addition, this planner could potentially be used with a 3D occupancy grid; a
use case for this is planning a goal point for a drone to fly around terrain

Figure 5: The path of the kart (green) over the true track centerline (red).

##  IV Conclusion

This paper presented a solution to the problem of traversing an unknown cone
defined race track within Autonomous racing. Our solution is beneficial for
the whole of autonomous racing because it provides a more simplistic and
generalized approach to navigating an unknown track. The algorithm presented
shows that utilizing an occupancy grid based method to navigate a race track
is feasible and robust. Additionally, the pipeline implemented here can be
extended further into more optimized solutions that can improve on what we
have shown. We are excited to see further developments and optimizations from
utilizing this pipeline within other existing and new autonomous racing
pipelines. The code for this algorithm can be found here.

##  V Acknowledgements

We would like to thank Kennesaw State University Electric Vehicle Team for
providing the opportunities, resources, and support to come up with this
pipeline. We would also like to thank the evGrandPrix organization and Purdue
University for their gracious dedication to this competition, in addition to
the other participating universities for their constant effort in growing and
shaping the community and challenges associated with this new competition.

## References

  * [1] S. J. Young, “Formula one racing: Driver vs. technology,” _Intersect: The Stanford Journal of Science, Technology and Society_ , vol. 5, 2012.
  * [2] A. Jiang, “Research on the development of autonomous race cars and impact on self-driving cars,” _Journal of Physics: Conference Series_ , vol. 1824, p. 012009, 03 2021.
  * [3] J. Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. Krovi, and R. Mangharam, “Autonomous vehicles on the edge: A survey on autonomous vehicle racing,” _IEEE Open Journal of Intelligent Transportation Systems_ , vol. 3, pp. 1–1, 01 2022.
  * [4] “Autonomous,” Dec 2021. [Online]. Available: https://engineering.purdue.edu/evGrandPrix/autonomous/
  * [5] “The ”disparity extender” algorithm, and f1/tenth,” Apr 2019. [Online]. Available: https://www.nathanotterness.com/2019/04/the-disparity-extender-algorithm-and.html
  * [6] L. Andresen, A. Brandemuehl, A. Hönger, B. Kuan, N. Vödisch, H. Blum, V. Reijgwart, L. Bernreiter, L. Schaupp, J. J. Chung, M. Bürki, M. R. Oswald, R. Y. Siegwart, and A. Gawel, “Accurate mapping and planning for autonomous racing,” _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 4743–4749, 2020.
  * [7] J. Jönsson and F. Stenbäck, “Monte-carlo tree search in continuous action spaces for autonomous racing: F1-tenth,” Ph.D. dissertation, 2020. [Online]. Available: http://urn.kb.se/resolve?urn=urn:nbn:se:hh:diva-42442
  * [8] S. Macenski, T. Foote, B. Gerkey, C. Lalancette, and W. Woodall, “Robot operating system 2: Design, architecture, and uses in the wild,” _Science Robotics_ , vol. 7, no. 66, p. eabm6074, 2022. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.abm6074
  * [9] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),” in _IEEE International Conference on Robotics and Automation (ICRA)_. Shanghai, China: IEEE, May 9-13 2011.
  * [10] G. Bradski, “The OpenCV Library,” _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * [11] G. Guennebaud, B. Jacob _et al._ , “Eigen v3,” http://eigen.tuxfamily.org, 2010.
  * [12] “Velodyne,” 2016. [Online]. Available: https://velodynelidar.com/products/puck/
  * [13] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” _Commun. ACM_ , vol. 24, no. 6, pp. 381–395, jun 1981. [Online]. Available: https://doi.org/10.1145/358669.358692
  * [14] R. C. Coulter, “Implementation of the pure pursuit path tracking algorithm,” 1992\.