# Building Hybrid B-Spline And Neural Network Operators

Raffaele Romagnoli, Jasmine Ratchford, and Mark H. Klein R. Romagnoli is with
the Department of Electrical and Computer Engineering, Carnegie Mellon
University, Pittsburgh, PA 15213, USA<EMAIL_ADDRESS>Ratchford and
M. H. Klein are with the Software Engineering Institute, Carnegie Mellon
university, Pittsburgh, PA 15235, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Control systems are indispensable for ensuring the safety of cyber-physical
systems (CPS), spanning various domains such as automobiles, airplanes, and
missiles. Safeguarding CPS necessitates runtime methodologies that
continuously monitor safety-critical conditions and respond in a verifiably
safe manner. A fundamental aspect of many safety approaches involves
predicting the future behavior of systems. However, achieving this requires
accurate models that can operate in real time. Motivated by DeepONets, we
propose a novel strategy that combines the inductive bias of B-splines with
data-driven neural networks to facilitate real-time predictions of CPS
behavior. We introduce our hybrid B-spline neural operator, establishing its
capability as a universal approximator and providing rigorous bounds on the
approximation error. These findings are applicable to a broad class of
nonlinear autonomous systems and are validated through experimentation on a
controlled 6-degree-of-freedom (DOF) quadrotor with a 12 dimensional state
space. Furthermore, we conduct a comparative analysis of different network
architectures, specifically fully connected networks (FCNN) and recurrent
neural networks (RNN), to elucidate the practical utility and trade-offs
associated with each architecture in real-world scenarios.

##  I Introduction

Control systems play a vital role in ensuring the safety of cyber-physical
systems (CPS), including automobiles, airplanes, and missiles [1]. During
design, mathematical techniques are employed to prevent these systems from
entering unsafe states. However, due to the dynamic interaction with the
physical environment, it is impossible to foresee every potential safety
hazard. Therefore, development-time methodologies often need to be
supplemented with runtime approaches [2] that continuously monitor safety-
critical conditions and react in a verifiably safe manner.

An approach to ensuring safety involves predicting the system’s future
behavior. However, this approach requires that model of behavior is accruate
and can be executed in real time. This is exceedingly challenging for complex
systems. The notable success of machine learning (ML) techniques has drawn
researchers’ attention towards scientific ML (SciML) methods, which can learn
from governing equations and data [3][4] and provide real-time solutions.
Scientific ML has been recently used to control applications for system
identification, model predictive control (MPC), analysis and verification, and
digital twins [5][6].

We are inspired by DeepONets [4], a neural network architecture strategy that
can be used to learn generalized non-linear operators. For an autonomous
system, these spaces could be considered the initial conditions, and the to a
unique solution (trajectory) that transports the autonomous system to a
desired end state. Although there are several different types of neural
operators, a primary advantage of DeepONets is their architecture, which is
primarily composed of separate neural networks that learn basis functions and
coefficients, enables the variation which we study in this paper.

A primary feature of DeepONets is that they learn basis functions for
representing the an approximate solution. This is an advantage for complex
systems where the basis functions are not easily described. However, it
introduces two places where the stochastic nature of statistical models
introduce uncertainty: the trunk net where the basis functions are learned and
in the branch net where the coefficients of the basis functions are learned.
In contrast, estimating trajectories from initial conditions, as might be
useful for autonomous systems safety analysis, have established, well-
researched, methods for describing continuous bases that impose an inductive
bias on the data-based learning - splines. In this paper we examine solutions
that combine neural operator theory with a learned basis function that
reflects a strong inductive bias. Specifically, we use B-splines as the basis
functions [7].

The key parameters that need to be learned when using B-splines to approximate
continuous functions are control points. We train a neural network to learn
the mapping from a set of differential equations’ initial conditions to the
control points of a B-spline approximation. The general approach of using
B-spline functions to approximate the solutions of PDEs in the field of Finite
Element Methods (FEMs) is referred to as isogeometric analysis (IGA). A
similar solution has been proposed in [8] where a neural network generates the
B-spline coefficients for a parametrized family of vector fields to reduce
computational costs.

This paper contributes by analyzing this approach within the framework of deep
neural operators, demonstrating its property as a universal approximator, and
providing bounds on the approximation error. These results are developed for a
general nonlinear autonomous system, and the approach is tested on a
controlled 6-degree-of-freedom (DOF) quadrotor with a state space of 12
dimensions. Additionally, a comparison between different network
architectures, namely fully connected neural networks (FCNN) and recurrent
neural networks (RNN), is provided.

A challenge in using ML-based approaches is proving bounds on the
uncertainties. The method proposed in this paper aims to open a new direction
towards this goal, which is fundamental in all aspects of CPSs, particularly
for safety. We distinguish our focus research research that focuses on
incorporating physics [6] for improved predictive performance and other
physics-based guarantees. In light of this, the choice of B-splines is
motivated by their interesting properties, such as the convex hull property,
where the predicted solution belongs to the convex hull generated by the
control points. This property can be beneficial for safety inference, as it
only requires checking a finite number of control points. These aspects
represent the next steps we aim to address in the near future along the
generalization to nonautonomous nonlinear systems.

##  II Preliminaries

This section is divided in the following parts: nonlinear systems [9], neural
networks [10], the universal approximation theorem for nonlinear operators
[11], B-splines functions [12] and finally the problem statement.

###  II-A Nonlinear Systems

Consider a nonlinear system expressed in the state-space form:

$\dot{x}=f(t,x)$ (1)

where $x\in\mathbb{R}^{n}$ is the state vector with $n$ components, and the
function $f:\mathbb{R}\times\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$ is
Lipschitz on a ball around the initial condition
$x_{0}\in\mathcal{B}_{r}=\left\\{x\in\mathbb{R}^{n}:\lVert x-x_{0}\rVert\leq
r\right\\}$ and for the time interval $[t_{0},t_{1}]$: $\lVert
f(x)-f(y)\rVert\leq L\lVert x-y\rVert$ for all $x,y\in\mathcal{B}_{r}$.
$\|\cdot\|$ is the Euclidean norm. From the local Cauchy’s theorem, there
exists a unique solution on the time interval $[t_{0},t_{0}+\delta]$ with a
positive $\delta\in\mathbb{R}$. According to Peano’s theorem, the solution can
be written as:

$x(t)=x_{0}+\int_{0}^{t}f(s,x(s))ds$ (2)

which is a continuous function $x:\mathbb{R}\rightarrow\mathbb{R}^{n}$. Let us
consider the space of continuous functions over the time interval $[a,b]$,
defined as $\mathcal{X}=C([a,b];\mathbb{R}^{n})$, which is a Banach space with
norm $\lVert x\rVert_{C}=\max_{t\in[0,\delta]}\lVert x(t)\rVert$. Considering
$\mathcal{S}=\left\\{x\in\mathcal{X}:\lVert x-x_{0}\rVert_{C}\leq r\right\\}$
it is possible to show that the right-hand side of (2) is a mapping
$\mathcal{P}:\mathcal{S}\rightarrow\mathcal{S}$, and moreover, this mapping is
a contraction:

$x(t)=\mathcal{P}(x)(t)$ (3)

where the function $x(t)$ is a fixed point.

We assume that the solution can be extended over the time interval $[0,T]$ and
the Lipschitz condition holds for a compact set $W\subset\mathbb{R}^{n}$.
Equation (1) is the general representation of a non-autonomous system, where
the explicit dependence on time $t$ can be due to an external input
$u\in\mathbb{R}^{p}$:

$\dot{x}=f(x,u)$

If the input signal is a state feedback control law of the form $u=\gamma(x)$,
where $\gamma:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n}$, the resulting closed-
loop system can be represented as an autonomous system of the form

$\dot{x}=f(x)$ (4)

Without loss of generality, we assume that $x={0}$ is an equilibrium point
(i.e., $f({0})={0}$) for (4), hence the origin is contained in $W$.

###  II-B Neural Networks

The fundamental element of a neural network (NN) is the standard neuron, which
takes $n$ inputs $\\{u_{1},u_{2},\ldots,u_{n}\\}$, where $u_{i}\in\mathbb{R}$,
and transforms them into a scalar value $y\in\mathbb{R}$. The activation
signal for the $i$-th neuron is computed as:

$s=u_{1}\theta_{i,1}+u_{2}\theta_{i,2}+\cdots+u_{n}\theta_{i,n}+\theta_{i,0}$
(5)

where $\theta_{i,j}\in\mathbb{R}$ with $j=1,\ldots,n$ are the weights that
need to be estimated, and $\theta_{i,0}$ is called the bias. The activation
signal (5) serves as the input to the activation function
$\sigma:\mathbb{R}\rightarrow\mathbb{R}$, which is typically nonlinear (e.g.,
sigmoidal, ReLU, etc.). Assuming we have a single-layer network with $N$
parallel neurons that use the same activation function, the output can be
expressed as:

$\begin{split}y&=\sum_{i=1}^{N}w_{i}\sigma\left(\sum_{j=1}^{n}\theta_{i,j}u_{j}+\theta_{i,0}\right)\\\
&=\sum_{i=1}^{N}w_{i}\sigma\left(\theta_{i}^{T}u+\theta_{i,0}\right)\end{split}$
(6)

where $\theta_{i}=[\theta_{i,1},\ldots,\theta_{i,n}]^{T}$,
$u=[u_{1},\ldots,u_{n}]^{T}$. If the output layer has a dimension greater than
one, (6) describes the $m$-th component of the output vector:

$y_{m}=\sum_{i=1}^{N}w_{i,m}\sigma\left({\theta_{i,m}}^{T}u+\theta_{i,0}^{m}\right)$
(7)

In the case of multi-layer networks known as feedforward neural networks
(FNNs), the output of one layer becomes the input of the next one:

$y^{[l+1]}_{m}=\sum_{i=1}^{N}w_{i,m}\sigma\left({\theta_{i,m}^{[l+1]}}^{T}y^{[l]}+\theta_{i,0}^{m^{[l+1]}}\right)$
(8)

where $[l+1]$ indicates the $(l+1)$-th layer, and $y^{[l]}$ is the
corresponding input vector, which is the output of the previous layer.
Equations (7) and (8) illustrate how a single-layer network with scalar output
can be extended to handle vector outputs and multi-layer FNNs. Consequently,
the theory of universal approximation of neural networks primarily focuses on
the case described in (6). In the following section, we provide basic
definitions and results on the universal approximation of nonlinear operators
with neural networks.

###  II-C The Universal Approximation Theorem for Nonlinear Operators

By considering a specific class of activation functions, Tauber-Wiener (TW)
functions, we present three main theorems from [11]. Theorems 1 and 2 deal
with the approximation of functions and functionals with NNs, respectively.
Finally, Theorem 3 combines the two previous results to approximate nonlinear
continuous operators.

###### Definition 1

A function $\sigma:\mathbb{R}\rightarrow\mathbb{R}$ is called a Tauber-Wiener
(TW) function if all linear combinations
$\sum_{i=1}^{N}c_{i}\sigma(\lambda_{i}z+\theta_{i})$,
$\lambda_{i},\theta_{i},c_{i}\in\mathbb{R}$, $i=1,2,...,N$ are dense in every
$C([a,b];\mathbb{R})$.

Let $g:K\rightarrow\mathbb{R}$, where $K\subset\mathbb{R}$ is a compact set,
$\mathcal{U}\subset C(K;\mathbb{R})$ is a compact set, and $\sigma\in(TW)$ is
an activation function.

###### Theorem 1

For any $\epsilon>0$, there exists a positive integer $N$,
$\theta_{0}^{i}\in\mathbb{R}$, $\theta_{i}\in\mathbb{R}^{n}$, $i=1,...,N$
independent of $g\in C(K;\mathbb{R})$, and constants $c_{i}(g)$, $i=1,...,N$
depending on $g$ such that

$\left|g(z)-\sum_{i=1}^{N}c_{i}(g)\sigma(\theta_{i}^{T}z+\theta_{0}^{i})\right|<\epsilon$
(9)

holds for all $z\in K$ and $g\in\mathcal{U}$. Moreover, each $c_{i}(g)$ is a
linear continuous functional defined on $\mathcal{U}$.

The last statement is due to the fact that in the proof, $c_{j}(g)$ represents
the Fourier coefficients of $g$. If $g:\mathbb{R}\rightarrow\mathbb{R}$ and
$K=[a,b]$, it can be associated with a specific time interval. Note that
$\mathbb{R}^{n}$ with $n\geq 1\in\mathbb{N}$ is a Banach space.

Now, let $\mathcal{X}$ be a Banach space, $\mathcal{K}\subseteq\mathcal{X}$,
$\mathcal{V}$ a compact set in $C(\mathcal{K};\mathbb{R})$, and $g$ a
continuous functional defined on $\mathcal{V}$.

###### Theorem 2

For any $\epsilon>0$, there exist a positive integer $N$, $m$ points
$z_{1},...,z_{m}\in\mathcal{K}$, and real constants $c_{j}$, $\zeta_{0}^{j}$,
$\xi_{i,j}$, $i=1,...,N$, $j=1,...,m$ such that

$\left|g(u)-\sum_{i=1}^{N}c_{i}\sigma\left(\sum_{j=1}^{m}\xi_{i,j}u(z_{j})+\zeta_{0}^{j}\right)\right|<\epsilon$
(10)

holds for all $u\in\mathcal{V}$.

Now, let $\mathcal{K}_{1}\subseteq\mathcal{X}$, $K_{2}\subseteq\mathbb{R}^{n}$
be two compact sets, $\mathcal{V}$ a compact set in
$C(\mathcal{K}_{1};\mathbb{R})$, and $\mathcal{G}$ a nonlinear continuous
operator mapping $\mathcal{V}$ into $C(K_{2};\mathbb{R})$.

###### Theorem 3

For any $\epsilon>0$, there exist positive integers $M$, $N$, $m$, constants
$c_{i}^{k}$, $\zeta_{0}^{jk}$, $\xi_{i,j}^{k}\in\mathbb{R}$, points
$\theta_{k}\in\mathbb{R}^{n}$, $z_{j}\in\mathcal{K}_{1}$, $i=1,...,M$,
$k=1,...,N$, $j=1,...,m$ such that

$\begin{split}&\left|\mathcal{G}(u)(y)-\sum_{k=1}^{N}\sum_{i=1}^{M}c_{i}^{k}\sigma\left(\sum_{j=1}^{m}\xi_{i,j}^{k}u(z_{j})+\zeta_{0}^{jk}\right)\right.\\\
&\hskip
128.0374pt\left.\cdot\sigma(\theta_{k}^{T}y+\theta_{0}^{k})\right.\Bigg{|}<\epsilon\end{split}$
(11)

holds for all $u\in\mathcal{V}$ and $y\in K_{2}$.

###  II-D B-splines

B-splines are piece-wise polynomial functions derived from slight adjustments
of Bezier curves, aimed at obtaining polynomial curves that tie together
smoothly. In this work, we are interested not in representing geometric
curves, but functions as in [12]. For this reason, we consider a parameter
$t\in K=[a,b]\subseteq\mathbb{R}$, and $(c_{i})_{i=1}^{\ell}\in\mathbb{R}$ as
a set of $\ell$ control points for a spline curve $s(t)$ of degree $d$, with
non-decreasing knots $(\hat{t}_{i})_{i=1}^{\ell+d+1}$.

$s(t)=\sum_{i=1}^{\ell}c_{i}B_{i,d}(t)\quad\text{for}\quad t\in
K\subseteq\mathbb{R},$ (12)

where $B_{i,d}(t)$, $d>1$, is given by the Cox-de Boor recursion formula [7]:

$B_{i,d}(t)=\frac{t-\hat{t}_{i}}{\hat{t}_{i+d}-\hat{t}_{i}}B_{i,d-1}(t)+\frac{\hat{t}_{i+d+1}-t}{\hat{t}_{i+d+1}-\hat{t}_{i+1}}B_{i+1,d-1}(t),$
(13)

and

$B_{i,0}(t)=\begin{cases}1,&\hat{t}_{i}\leq t<\hat{t}_{i+1},\\\
0,&\text{otherwise}.\end{cases}$ (14)

Property 1: Any value assumed by $s(t)$, $\forall t\in K$, lies in the convex
hull of its $\ell+1$ control points $(c_{i})_{i=1}^{\ell}$.

Property 2: Suppose that the number $\hat{t}_{i+1}$ occurs $m$ times among the
knots $(\hat{t}_{j})_{j=i-d}^{m+d}$ with $m$ an integer bounded by $1\leq
m\leq d+1$, e.g.,
$\hat{t}_{i}<\hat{t}_{i+1}=\cdots=\hat{t}_{i+m}<\hat{t}_{i+m+1}$, then the
spline function $s(t)$ has a continuous derivative up to order $d-m$ at knot
$\hat{t}_{i+1}$.

This property implies that the smoothness of the spline can be adjusted using
multiple knot points. A common choice is to set $m=d+1$ multiple knot points
for the initial and final knot points. This way, Equation (12) assumes the
first and final control points as initial and final values.

Property 3: the B-spline basis functions are continuous in $t$, $B_{i,d}(t)\in
C(K;\mathbb{R})$, and bounded [13].

By defining the vectors

$\begin{split}c&\triangleq[c_{1}\,c_{2}\,\dots\,c_{{\ell}}]^{T}\\\
B_{d}(t)&\triangleq[B_{1,d}(t)\,B_{2,d}(t)\,\dots\,B_{\ell,d}(t)]\end{split}$
(15)

(12) can be rewritten as $s(t)=B_{d}(t)c$.

B-splines are generally used to represent curves. However, in our problem, we
require a function of time that can be conceptualized as a 2D curve.
Therefore, each control point should be a 2D point. To ensure that $s(t)$ is
indeed a function of time, we associate the parameter $t$ with the time
variable that varies within the interval $[a,b]$. We enforce that the position
of each control point along the axis representing time $t$ remains fixed by
partitioning the interval $[a,b]$ into $\ell-1$ equispaced sub-intervals. By
doing so, we obtain the formulation presented in (15), where each control
point is associated with only a scalar coefficient.

For the purpose of this paper we are interested to the B-spline representation
for the approximation of the solution of (4) with $x\in\mathbb{R}^{n}$
consists of a set of $n$ scalar B-spline, one for each component of the state
vector. In general this B-spline is represented as

$\underline{s}(t)=\underline{B}_{d}(t)\underline{c}$ (16)

where $\underline{s}:\mathbb{R}\rightarrow\mathbb{R}^{n}$, and

$\displaystyle\underline{B}_{d}(t)$ $\displaystyle\triangleq$
$\displaystyle\mathrm{diag}[B_{d}(t)]$ (17) $\displaystyle\underline{c}$
$\displaystyle\triangleq$
$\displaystyle\left[\underline{c}_{1}^{T},\cdots,\underline{c}_{n}^{T}\right]^{T}$
(18) $\displaystyle\underline{c}_{i}$ $\displaystyle\triangleq$
$\displaystyle\left[c_{i1},\cdots,c_{i\ell}\right]^{T}.$ (19)

The index $i$ indicates the $i$-th components of the multidimensional B-spline
$\underline{s}$.

###  II-E Problem Statement

In this work, our focus is on providing future state estimation of nonlinear
closed-loop systems of the form (4) by approximating the operator
$\mathcal{P}(x)(t)$ (3) via a deep operator network that combines deep neural
networks and B-spline functions. By momentarily assuming that (4) is scalar,
our goal is to approximate $\mathcal{P}(x)(t)$ as follows:

$x(t)=\mathcal{P}(x)(t)\approx\hat{\mathcal{P}}(x)(t)=\sum_{i=1}^{\ell}c_{i}(x_{0})B_{i,d}(t)$
(20)

for $t\in[0,T]$, where the coefficients $c_{i}(x_{0})$ represent the
B-spline’s control points along the time interval $[0,T]$, determining the
shape of $x(t)$ depending on the initial condition $x_{0}$. The $c_{i}(x_{0})$
is a functional as depending of the initial condition $x_{0}$ which is
approximated by a deep neural network. Building upon the theoretical framework
established in [11], we extend the theory to encompass (20). Subsequently, we
delve into the multidimensional scenario, where $x\in\mathbb{R}^{n}$,
particularly when addressing error bounds.

##  III B-spline-Based Deep Neural Operator

Let us consider the same conditions of Theorem 3, modified for the problem of
finding an approximating operator (20) for the scalar case. We consider
$\mathcal{X}=C([0,T];\mathbb{R})$ as a Banach space with the norm
$\|x\|_{C}=\max_{t\in[0,T]}\|x(t)\|$. Let $\mathcal{S}\subseteq\mathcal{X}$ be
a compact set, and the operator $\mathcal{P}(x)(t)$ maps an element of
$\mathcal{S}$ into itself. The scalar initial condition
$x_{0}=x(0)\in\mathcal{S}$.

###### Theorem 4

For any $\epsilon>0$, there exist positive integers $M$, $\ell$, constants
$c_{i}^{k}$, $\zeta_{0}^{jk}$, $\xi_{i}^{k}\in\mathbb{R}$, functions
$B_{k,d}(t)\in C([0,T],\mathbb{R})$, and $x_{0}\in\mathcal{S}$ such that

$\left|\mathcal{P}(x)(t)-\sum_{k=1}^{\ell}\sum_{i=1}^{M}c_{i}^{k}\sigma\left(\xi_{i}^{k}x_{0}+\xi_{0}^{ik}\right)\cdot
B_{k,d}(t)\right|\leq\epsilon$ (21)

holds for all $x_{0}\in\mathcal{S}$ and $t\in K_{2}$.

###### Proof:

To prove this theorem, we follow a similar approach used in [4]. In this case,
the branch network
$\sum_{i=1}^{M}c_{i}^{k}\sigma\left(\xi_{i}^{k}x_{0}+\xi_{0}^{ik}\right)$ is
expressed in the same form as in Theorem 3, except the trunk network is
replaced by the B-spline basis functions $B_{k,d}(t)$. Recalling the proof of
Theorem 3 (Theorem 5 in [11]), we need to show that we can write

$\left|\mathcal{P}(x)(t)-\sum_{k=1}^{\ell}c_{k}(\mathcal{P}(x))B_{k,d}(t)\right|\leq\epsilon/2$
(22)

Theorem 1 cannot be used since $B_{k,d}(t)$ is a polynomial function. Instead,
we can use the Weierstrass approximation theorem [14]. This theorem is proven
by using a linear combination of Bernstein polynomials used as basis
functions. This linear combination of polynomials is also used to generate
Bézier curves which are a particular subtype of B-spline functions [15]. In
particular, it is possible to transform a Bézier curve into a B-spline and
vice-versa [13], therefore we can switch from a representation with B-spline
basis function $B_{k,d}(t)$ to Bernstein polynomials and vice-versa. This
implies that we can use B-splines as universal approximators of functions, and
for this reason, (22) holds. ∎

For the case of $x\in\mathbb{R}^{n}$, it is possible to redefine Theorem 4 as
Theorem 2 in [4]. Specifically, we need to define one B-spline for each
component of the state, and the input to the network is a vector
$x_{0}\in\mathbb{R}^{n}$, where for each neuron $i$, we have
$\sigma\left(\sum_{j=1}^{n}\xi_{i,j}^{k}x_{0}^{j}+\zeta_{0}^{ik}\right)$.

###  III-A Initial condition and control points mapping

Let us assume that for a specific initial condition $x_{0}\in
W\subseteq\mathbb{R}$ we compute the particular solution $x(t)$ for
$t\in[0,T]$. We can take $N$ samples of the solution $x(t)$ by properly
choosing a sampling step $h$ and we can form a vector of measurements
$y_{m}=[x(0),x(h),x(2h),...,x((N-1)h)]^{T}$. From (15), we can write

$y_{m}=C_{m}c_{x_{0}};\qquad C_{m}=\left[\begin{array}[]{c}B_{d}(0)\\\
B_{d}(h)\\\ \vdots\\\
B_{d}((N-1)h)\end{array}\right]\in\mathbb{R}^{N\times\ell}.$ (23)

The vector of control points defined in (15) is renamed here as $c_{x_{0}}$ to
indicate the solution to a specific trajectory $x(t)$ generated from $x_{0}$.
From the B-spline properties, the rank of $C_{m}$ is equal to $\ell$, then it
is possible to find the vector of control points $c_{x_{0}}$ by using the
pseudo inverse $c_{x_{0}}=(C_{m}^{T}C_{m})^{-1}C_{m}^{T}y_{m}$ which minimizes
$\|y_{m}-C_{m}c_{x_{0}}\|^{2}$. As a consequence of the Weierstrass
approximation theorem, it follows that by appropriately choosing values for
$\ell$ and $h$, the error can be reduced to a desired level.

$|x(t)-B_{d}(t)c_{x_{0}}|<\epsilon.$

For each initial condition $x_{0}\in W$ we can find a unique sequence of
control point since the initial control point corresponds with the initial
condition itself. This means that there is a mapping $\mathcal{M}$ that
associates to each initial condition $x_{0}\in W$ a sequence of $\ell$ control
points. From Theorem 4 it follows that

$\mathcal{M}(x_{0})\approx\hat{\mathcal{M}}(x_{0})=\left[\begin{array}[]{c}\sum_{i=1}^{M}c_{i}^{1}\sigma(\xi_{i}^{1}x_{0}+\zeta_{0}^{i1})\\\
\vdots\\\
\sum_{i=1}^{M}c_{i}^{\ell}\sigma(\xi_{i}^{\ell}x_{0}+\zeta_{0}^{i\ell})\end{array}\right]$
(24)

where the neural network is used to approximate the mapping $\mathcal{M}$. In
this way an infinite dimensional problem has been reduced to a finite one. The
use of B-splines guarantees that the approximation of the solution of $x(t)$
is a continuous function defined for any $t\in[0,T]$. In the next subsection
we define the problem for the multidimensional case $x\in\mathbb{R}^{n}$,
formalize the existence of the mapping $\mathcal{M}$, and provide some error
bounds.

###  III-B Multidimensional Solution and Error Bounds

For the general problem where $\hat{\mathcal{P}}(x)$ approximates the solution
of $n$ dimensional ODEs such as (4), (20) can be written as

$\hat{\mathcal{P}}(x)(t)=\underline{B}_{d}(t)\hat{\mathcal{M}}(x_{0})$ (25)

where $\underline{B}_{d}(t)\in\mathbb{R}^{n\times(n\cdot\ell)}$ accordingly
with (16), and the network
$\hat{\mathcal{M}}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\cdot\ell}$ has $n$
inputs and $n\cdot\ell$ outputs.

###### Lemma 1

For the autonomous system described by (4), where $x_{0}$ denotes the initial
condition in $W\subset\mathbb{R}^{n}$, and $x(t)$ denotes a solution for the
time interval $[0,T]$, with $h>0$ representing a sampling time and $\ell>0$
indicating the number of B-spline control points for each component of the
state $x$, there exists a mapping
$\mathcal{M}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{n\cdot\ell}$ such that

$\|x(t)-\underline{B}_{d}(t)\mathcal{M}(x_{0})\|\leq\epsilon$ (26)

where, $\underline{B}_{d}(t)\in\mathbb{R}^{n\times(n\cdot\ell)}$.

###### Proof:

For each $i$-th component of a solution $x(t)$ obtained for a specific initial
condition $x_{0}\in W$, we can formulate the least squares problem (23), where
$y_{m,i}=C_{m}c_{i,x_{0}}$. Here, we define:

$\underline{y}_{m}=\left[y_{m,1}^{T},\cdots,y_{m,n}^{T}\right],\;\underline{C}_{m}=\mathrm{diag}[C_{m}]$

and
$\underline{c}_{x_{0}}=\left[{c}_{1,x_{0}}^{T},\cdots,{c}_{n,x_{0}}^{T}\right]^{T}$.
This problem assumes the least squares form
$\underline{y}_{m}=\underline{C}_{m}\underline{c}_{x_{0}}$, which can be
solved using the pseudo-inverse. As $\underline{C}_{m}$ is a block diagonal
matrix where each block has $\ell$ independent columns, a unique sequence of
control points is generated by the least squares. Consequently, there exists a
mapping $\mathcal{M}$ that maps any $x_{0}$ into the B-spline control points
domain, ensuring (26) holds. ∎

Similarly to the approach in [16], where a FNN is employed to approximate the
one-step evolution function, we utilize a generalization of Theorem 1 in [17]
to establish the following proposition:

###### Proposition 1

Considering the conditions of Lemma 1 and a mapping $\mathcal{M}$ satisfying
(26), there exists an FNN $\hat{\mathcal{M}}$ such that

$\|\mathcal{M}(\cdot)-\hat{\mathcal{M}}(\cdot)\|\leq\gamma$ (27)

for any $x_{0}\in W$.

We define $\hat{x}(t)\triangleq\underline{B}_{d}(t)\hat{\mathcal{M}}(x_{0})$
and $\tilde{x}(t)\triangleq\underline{B}_{d}(t){\mathcal{M}}(x_{0})$, where
$\mathcal{M}$ and $\hat{\mathcal{M}}$ retain their meanings from Proposition
1.

###### Lemma 2

Under the conditions of Lemma 1, for the problem of finding
$\hat{\mathcal{P}}(x)$, there exists $M>0$ such that

$\|x(t)-\hat{x}(t)\|\leq\epsilon+M\gamma$ (28)

for any $t\in[0,T]$, where $\epsilon$ and $\gamma$ are the bounds related to
the least squares and FNN approximation, respectively.

###### Proof:

For any $t\in[0,T]$, we can write

$\begin{split}\|x(t)-\hat{x}(t)\|&=\|x(t)-\hat{x}(t)+\tilde{x}(t)-\tilde{x}(t)\|\\\
&\leq\|x(t)-\tilde{x}(t)\|+\|\tilde{x}(t)-\hat{x}(t)\|\end{split}$ (29)

From (26) of Lemma 1, we have $\|x(t)-\tilde{x}(t)\|\leq\epsilon$, whereas

$\begin{split}\|\tilde{x}(t)-\hat{x}(t)\|&=\|\underline{B}_{d}(t)(\mathcal{M}(x_{0})-\hat{\mathcal{M}}(x_{0}))\|\\\
&\leq\|\underline{B}_{d}(t)\|\|\mathcal{M}(x_{0})-\hat{\mathcal{M}}(x_{0})\|\leq
M\gamma\end{split}$ (30)

where $\gamma$ results from Proposition 1, and $M$ results from the
boundedness of the B-spline basis functions [13]. ∎

##  IV Experiments

We demonstrate the value of this technique through several experiments with a
quadrotor with nonlinear dynamics and controlled with a linear quadratic
regulator (LQR) controller [18]. The input in every experiment was the
12-dimensional initial condition, consisting of the 3-D position, velocity,
angular orientation, and angular velocity. The output is a set of control
points given a known sampling interval and time horizon.

We generated data by randomly choosing initial conditions within a 12-D ball
limited by 2 meters in each spatial dimension, 2 meters per second in each
velocity dimension, $\pi/4$ radians in each angular dimension and 5 radians
per second in each angular velocity dimension. All generated trajectories
assumed a desired equilibrium point at $\vec{0}$ and we refer to it as
$x_{eq}$. For each trajectory, the controls points that define each trajectory
were calculated using least squares fitting and a 3rd order B-spline ($d=3$).
For all experiments we used 5000 initial conditions for training. To augment
our dataset and encourage rotational equivariance, we rotated our randomly
generated initial conditions by $\pi$, $-\pi/2$, $-\pi/4$, $-\pi/6$,
$\pi/4$,$\pi/3$, $\pi/2$ radians around the Z axis.

In our first experiment, we set the number of control points to 50, using a
time horizon of 2.5 seconds. We used a simple fully-connected neural network
(FCNN) with 12 layers. Each hidden layer has a width of 120 neurons and the
input and output layers are dictated by the space dimension (12) and number of
control points (50). Therefore the output layer has a width of 600. We
optimized over the mean squared error between the predicted sampled trajectory
and the trajectory found through least squares fitting. In order to examine
model variance, training was reinitialized 10 times and training was stopped
after 2 hours or when the loss had reached about $10^{-5}$,which ever
condition was satisfied first. One model out of the 10 attempts did not
converge within this loss condition and, thus, was not used in the subsequent
analysis.

A FCNN does not exploit the physical properties of the system. In our second
experiment we combined a gated recurrent unit (GRU) in combination with a FCNN
in a recurrent neural network (RNN) architecture. Both the GRU and the
subsequent fully connected layers had a width of 120. The fully connected
layers had a depth of 3. This resulted in a lower validation loss (mean
squared error) for fewer than half as many parameters, as can be seen in Table
I.

##  V Results

We used the trajectories and control points found using least squares fitting
via singular value decomposition with Python and Numpy as as ground truth in
training the neural networks. In order to evaluate and compare across our
experiments, we evaluated against a test set containing 1000 new initial
conditions. In Table I we compare the timing between different experiments and
the ODE Solver We find that the FCNNs is 5 times faster than least squares
fitting and the RNN is 5 times slower than SciPy ODE solver. The increase in
evaluation time between the FCNN and the RNN is a consequence of a for-loop
required to calculate the entire trajectory. Notably, the ODE solver used is a
python wrapper around a ODEPACK library whereas the neural networks were built
using PyTorch. Therefore, we advise caution when comparing these evaluation
times.

TABLE I: Time comparisons for different methods. Method | T[sec] | $\ell$ | # Parameters | Mean Final MSE (Loss) | # Models | Evaluation Time [sec]
---|---|---|---|---|---|---
ODE solver | 2.5 | 50 | - | - | - | $0.0049\pm 0.0015$
FCNN | 2.5 | 50 | 219360 | $1.4\pm 0.2\times 10^{-5}$ | 9 | $0.00097\pm 0.00023$
RNN | 2.5 | 50 | 78732 | $2.9\pm 0.6\times 10^{-6}$ | 10 | $0.028\pm 0.003$

In both cases, the error is significantly correlated with the initial
condition, as parameterized by the distance in the 12-D ball of the initial
condition. This can be seen in Figure 1. This result is aligned with several
factors: 1) Since $x_{eq}$ is asymptotically stable, the radius of the initial
condition in the unit ball serves as a measure of the distance the quadrotor
is from it; 2) Our initial conditions were randomly sampled within the cube of
the radius to balance comprehensive sampling throughout the volume and
sampling at different radii. Consequently, the density of training points
decreases with the radius.

Figure 1:  (top) FCNN and (bottom) RNN root mean squared error versus initial
condition radius. The error is positively correlated with distance.The overall
error can be seen in the profile histogram right of the scatter plot.

Because summary statistics can only provide a partial understanding of our
prediction accuracy, we chose four specific initial conditions to analyze
qualitatively by varying the initial condition in a single dimension. These
are shown in Figure 2. The initial conditions we considered were (with all
unspecified values set to zero):

  * •

$x^{y}_{0}=(0.8\,m,1.2\,m)$ \- Red (solid and dotted) lines indicate the
least-squares (”ground truth”) trajectories and the pink shading indicates the
range of the respective neural-network approximated trajectories.

  * •

$x^{y}_{0}=1.0\,m,x^{dy}_{0}=(0.8\;m/s,1.2\,m/s)$ \- Blue lines and shaded
regions.

  * •

$x^{y}_{0}=-1.0\,m,x^{d\theta}_{0}=(4\,rad/s,6\;rad/s)$ \- Green lines and
shaded regions.

  * •

$x^{y}_{0}=-1.0\,m,\;x^{d\theta_{y}}_{0}=5\,rad/s,x^{\theta_{y}}_{0}=\pi/16\cdot(3,5)$
\- Orange lines and shaded regions.

  * •

$x_{0}=R/2$ where R is the maximum value in any dimension, except for
$x_{z}=-R/2$. Black lines and gray shaded regions. In this instance, all
directions were varied slightly.

Figure 2:  (left) FCNN and (right) RNN predicted and least-squares fitted
trajectories for sets of initial condition, varied about a central value as
indicated on the legend. Only four dimensions are shown to conserve space.

This provides anecdotal evidence of the per-trajectory fit and shows how
variation due to the neural network architecture and training compare to
variation in the initial values. The variation due to initial condition
variation is present in both the neural network approximations (shaded
regions) and least squares fit of spline control points to the solution found
with an ODE solver (lines). However, the shaded regions depicting model
variations have uncertainty contributions from both initial conditions and
model stochasticity. Notably, for both the RNN and the FCNN, the variance in
initial conditions dominates except for the final case where all we dimensions
are significantly non-zero. This agrees with the earlier assessment that the
primary indicator of uncertainty is the proximity of the initial condition to
the edge of the ball of potential initial conditions. Furthermore, excepting
the same case, the least squares trajectory is always within the trajectories
indicated by the neural network. Note that only four of twelve dimensions are
shown for brevity.

If the resulting neural operator (neural network plus b-spline system)
reflects the invariances and symmetries of the actual system, there are
stronger guarantees that the neural operator will generalize to new, unseen
conditions. In Figure 3, we show the difference root mean squared difference
between our original testing dataset and the test dataset rotated to 85 evenly
spaced angles between $\pi/8$ and $15\pi/8$ and the resulting trajectory,
rotated back. An ideal equivariant operator has the property
$R_{\theta}^{T}\mathcal{P}(R_{\theta}x_{0})=R_{\theta}^{T}R_{\theta}\mathcal{P}(x_{0})=\mathcal{P}(x_{0})$
and the root mean squared difference,
$\langle(R_{\theta}^{T}\mathcal{P}(R_{\theta}x_{0})-\mathcal{P}(x_{0}))^{2}\rangle^{1/2}=0$.
We find that this deviation is comparable to the overall error exhibited by
the respective neural networks. Notably, the RNN has a long low-RMSE tail. We
suggests that this indicates an recurrent or residual architecture is more
well adapted to this problem and we may be able to improve accuracy with a
refined recurrent or residual network architecture.

Figure 3: The initial condition is rotated around the Z axis to 85 different
angles from $\pi/8$ to $15\pi/8$. The (top) FCNN or (bottom) RNN is applied.
The root mean squared difference is compared to the radius of the initial
condition in the 12-D ball.

##  VI Conclusions

We are exploring various improvements and adjustments to the architecture and
training of the neural network that could enhance performance. Notably, neural
architectures like residual blocks naturally capture the iterative nature of
sequential control points. Additionally, incorporating a long-short-term
memory cell (LSTM) could enhance temporal smoothness. Furthermore, embedding
the rotational symmetries of the quadrotor into the neural network design
using equivariant network structures for dynamical systems [19, 20], such as
group equivariant convolutional neural networks or SympNets [21, 22], may also
improve performance.

In addition to enhancing the contributions presented in this paper, our next
endeavor involves developing a framework to estimate and fine-tune the neural
network errors by leveraging the B-spline convex hull property. Ensuring that
the actual state trajectory resides within the convex hull of the control
points enables real-time evaluations of safety violations. Furthermore, a
crucial aspect of our future work entails extending this framework to non-
autonomous systems, which is pivotal for advancing the development of neural
network-based controllers.

##  VII acknowledgements

Copyright 2024 Carnegie Mellon University and Raffaele Romagnoli

This material is based upon work funded and supported by the Department of
Defense under Contract No. FA8702-15-D-0002 with Carnegie Mellon University
for the operation of the Software Engineering Institute, a federally funded
research and development center.

[DISTRIBUTION STATEMENT A] This material has been approved for public release
and unlimited distribution. Please see Copyright notice for non-US Government
use and distribution. DM24-0319

## References

  * [1] A. Platzer, Logical Foundations of Cyber-Physical Systems.  Springer, 2018.
  * [2] D. de Niz, B. Andersson, M. Klein, J. Lehoczky, A. Vasudevan, H. Kim, and G. Moreno, “Mixed-trust computing for real-time systems,” in 2019 IEEE 25th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), pp. 1–11, IEEE, 2019.
  * [3] N. Baker, F. Alexander, T. Bremer, A. Hagberg, Y. Kevrekidis, H. Najm, M. Parashar, A. Patra, J. Sethian, S. Wild, et al., “Workshop report on basic research needs for scientific machine learning: Core technologies for artificial intelligence,” tech. rep., USDOE Office of Science (SC), Washington, DC (United States), 2019.
  * [4] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis, “Learning nonlinear operators via deeponet based on the universal approximation theorem of operators,” Nature machine intelligence, vol. 3, no. 3, pp. 218–229, 2021.
  * [5] T. X. Nghiem, J. Drgoňa, C. Jones, Z. Nagy, R. Schwan, B. Dey, A. Chakrabarty, S. Di Cairano, J. A. Paulson, A. Carron, et al., “Physics-informed machine learning for modeling and control of dynamical systems,” in 2023 American Control Conference (ACC), pp. 3735–3750, IEEE, 2023.
  * [6] A. Saviolo, G. Li, and G. Loianno, “Physics-inspired temporal learning of quadrotor dynamics for accurate model predictive trajectory tracking,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10256–10263, 2022.
  * [7] C. De Boor and C. De Boor, A practical guide to splines, vol. 27.  springer-verlag New York, 1978.
  * [8] K. Doległo, A. Paszyńska, M. Paszyński, and L. Demkowicz, “Deep neural networks for smooth approximation of physics with higher order and continuity b-spline base functions,” arXiv preprint arXiv:2201.00904, 2022.
  * [9] H. K. Khalil, Nonlinear systems; 3rd ed. Upper Saddle River, NJ: Prentice-Hall, 2002.
  * [10] S. Bittanti and G. Picci, Identification, adaptation, learning: the science of learning models from data, vol. 153.  Springer Science & Business Media, 1996.
  * [11] T. Chen and H. Chen, “Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems,” IEEE transactions on neural networks, vol. 6, no. 4, pp. 911–917, 1995.
  * [12] L. Jetto, V. Orsini, and R. Romagnoli, “B-splines and pseudo-inversion as tools for handling saturation constraints in the optimal set-point regulation,” in 2017 American Control Conference (ACC), pp. 1041–1048, IEEE, 2017.
  * [13] H. Prautzsch, W. Boehm, and M. Paluszny, Bézier and B-spline techniques, vol. 6.  Springer, 2002.
  * [14] K. R. Davidson and A. P. Donsig, Real Analysis and Applications: Theory in Practice.  Springer New York, NY, 2010.
  * [15] F. Chudy and P. Woźny, “Linear-time algorithm for computing the bernstein–bézier coefficients of b-spline basis functions,” Computer-Aided Design, vol. 154, p. 103434, 2023.
  * [16] T. Qin, Z. Chen, J. D. Jakeman, and D. Xiu, “Data-driven learning of nonautonomous systems,” SIAM Journal on Scientific Computing, vol. 43, no. 3, pp. A1607–A1624, 2021.
  * [17] A. Pinkus, “Approximation theory of the mlp model in neural networks,” Acta numerica, vol. 8, pp. 143–195, 1999.
  * [18] R. Romagnoli, B. H. Krogh, D. de Niz, A. D. Hristozov, and B. Sinopoli, “Software rejuvenation for safe operation of cyber–physical systems in the presence of run-time cyberattacks,” IEEE Transactions on Control Systems Technology, 2023.
  * [19] S. Villar, D. W. Hogg, K. Storey-Fisher, W. Yao, and B. Blum-Smith, “Scalars are universal: Equivariant machine learning, structured like classical physics,” in Advances in Neural Information Processing Systems (M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, eds.), vol. 34, pp. 28848–28863, Curran Associates, Inc., 2021.
  * [20] R. Wang, R. Walters, and R. Yu, “Incorporating Symmetry into Deep Dynamics Models for Improved Generalization,” Mar. 2021.
  * [21] D. Romero, E. Bekkers, J. Tomczak, and M. Hoogendoorn, “Attentive group equivariant convolutional networks,” in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 8188–8199, PMLR, 13–18 Jul 2020.
  * [22] P. Jin, Z. Zhang, A. Zhu, Y. Tang, and G. E. Karniadakis, “Sympnets: Intrinsic structure-preserving symplectic networks for identifying hamiltonian systems,” Neural Networks, vol. 132, pp. 166–179, 2020.


# Self-supervised Egomotion and Depth Learning via Bi-directional Coarse-to-
Fine Scale Recovery

Hao Qu, Lilian Zhang, Xiaoping Hu, Xiaofeng He, Xianfei Pan, Changhao Chen The
authors are with the College of Intelligence Science and Technology, National
University of Defense Technology, Changsha, 410073, China*Corresponding
author: Changhao Chen (changhao.chen66@outlook.com)This work was supported by
National Natural Science Foundation of China (NFSC) under the Grant Number of
62103427, 62073331, 62103430.

###### Abstract

Self-supervised learning of egomotion and depth has recently attracted great
attentions. These learning models can provide pose and depth maps to support
navigation and perception task for autonomous driving and robots, while they
do not require high-precision ground-truth labels to train the networks.
However, monocular vision based methods suffer from pose scale-ambiguity
problem, so that can not generate physical meaningful trajectory, and thus
their applications are limited in real-world. We propose a novel self-learning
deep neural network framework that can learn to estimate egomotion and depths
with absolute metric scale from monocular images. Coarse depth scale is
recovered via comparing point cloud data against a pretrained model that
ensures the consistency of photometric loss. The scale-ambiguity problem is
solved by introducing a novel two-stages coarse-to-fine scale recovery
strategy that jointly refines coarse poses and depths. Our model successfully
produces pose and depth estimates in global scale-metric, even in low-light
condition, i.e. driving at night. The evaluation on the public datasets
demonstrates that our model outperforms both representative traditional and
learning based VOs and VIOs, e.g. VINS-mono, ORB-SLAM, SC-Learner, and UnVIO.

###### Index Terms:

Visual Odometry, Depth Learning, Visual-inertial Odometry, Deep Neural
Netework

##  I Introduction

Egomotion and scene perception plays a vital role in autonomous systems. Self-
driving and mobile robots are normally equipped with a combination of sensors,
e.g. camera, inertial measurement unit (IMU) and LIDAR, for self-motion and
scene perception, which further supports path planning and decision making.

A representative method is visual odometry (VO), which tracks and matches
visual features between consecutive images, and establishes multi-view
geometry model to calculate relative pose. However, pose estimates from
monocular VO are scale-ambiguous. Stereo VO exploits stereo camera to recover
the absolute scale of pose estimates, and is generally more robust because of
information fusion from two cameras. Visual-inertial odometry (VIO) attracts
attentions, as it combines camera with inertial sensor to estimate system
states that can recover global-scale, and improve the robustness in complex
environments.

Recently, deep learning based methods construct end-to-end deep neural network
model to learn pose from images directly. They can be generally categorized
into supervised learning based and unsupervised learning based approaches.
Supervised learning based VO/VIOs require ground-truth pose as labels to train
the neural network model. Though they show good performance in challenging
scenes, e.g. in featureless areas or in complex light-conditions, labelling
data is costly and time-consuming. In addition, the trained model is hard to
generalize to new environments with completely different scene geometry and
appearance. Thus, there are a number of attempts on unsupervised learning
based VO/VIOs that can jointly learn to produce relative pose and scene
depths, by constructing photometric loss between the real image and wrapped
image from novel view synthesis. However, They still suffer from scale
ambiguity problem.

To solve this problem, previous researchers use prior information, e.g.
velocity or camera height to recover the scale metric. A recent work, i.e.
VIOLearner [1], calculates the interpolated depths from pointcloud data to
replace depth learning, and use the depths with absolute scale to obtain pose
with absolute scale. However, the performance of scale recovery highly depends
on the quality of pointcloud data, i.e. sparsity. Moreover, the interpolated
depths do not conform to the multi-view geometry, and thus can be not used as
supervised labels for depth learning, so that VIOLearner only produces pose
estimates.

We instead propose a self-supervised learning framework to jointly estimate
ego-motion and depths with global scale metric from image sequences, via a
novel Bi-directional coarse-to-fine scale recovery process. Instead of
interpolating depths directly, in our model, point clouds data are used to
compare with a pretrained depth model to calculate coarse scale ratio and
depths. A two-stages process of scale recovery is proposed to firstly produce
coarse pose via a single-directional scale recovery conditioned on coarse
depths, and then refine depths and poses together through a bi-directional
coarse-to-fine scale recovery. Our model is capable of generating both depths
and poses with global scale accurately and robustly, even in low-light
conditions (i.e. driving at night). In addition, inertial data can also be
introduced into the framework via an attention based fusion module that
further improves the performance of rotation estimation.

Our contributions can be summarized as follows:

  * •

We propose a novel self-supervised learning framework that can jointly
estimate ego-motion and depths with global scale metric from image sequences.

  * •

We propose a novel Bi-directional coarse-to-fine scale recovery process that
jointly refines coarse poses and depths. Depth scale is recovered via
comparing point cloud data against a pretrained model that ensures the
consistency of photometric loss.

  * •

Extensive experiments were conducted to validate the effectiveness of our
proposed model. Our model produces pose and depth estimates robustly, even in
low-light condition (i.e. a driving scene at night), a very challenging task
for traditional VO/SLAM.

##  II Related Works

###  II-A Traditional Visual Odometry

Figure 1: overview of our framework

Visual Odometry (egomotion estimation) is to calculate the relative pose of a
moving camera through the matching feature points between different image
frames via multi-view geometry model. Its main processes can be divided into:
feature point extraction, feature point matching, outlier elimination, and
bundle adjustment. LIBVISO[2] is a classic model-based visual odometry, that
uses two different types of convolution kernels to detect the maximum response
points in the image as feature points, and uses RANSAC[3] algorithm to filter
the matching feature points. PTAM[4] uses FAST [5] algorithm to detect feature
points on images with different resolutions, in order to improve the
performance of odometry at different scales. ORB-SLAM [6] adds stereo camera
and depth camera to extend its application. In the texture-missing
environment, the direct-method shows better robustness in pose estimation than
feature-point based methods. DSO[7] and LSD-SLAM[8] use the corresponding
pixel difference between adjacent images to construct photometric loss model
to calculate camera pose. Under low-light environments, the performance of
visual odometry decreases due to the failure of feature dection and matching.
Combined IMU with camera, visual-inertial odometry (VIO) obtains better
robustness and higher accuracy in pose estimation. [9] uses IMU pre-
integration to integrate IMU raw data and optimizes the re-projection loss
using matching feature points and point clouds. VINS[10] adds automatic
initialization, relocation, loop closure detection and other functions to form
a more complete visual-inertial odometry.

###  II-B The supervised learning of ego-estimation estimation

DeepVO[11] is a representative work of end-to-end learning based visual
odometry (VO). It uses the convolution layers of FlowNet [12] as the visual
feature extractor, and a LSTM[13] network to integrate the visual features of
multiple images. Fully-connected layers are added after the LSTM network to
transform features into pose predictions. Leveraging high-accuracy pose labels
to train the model, DeepVO can learn to predict absolute pose predictions.
[14] uses the LSTM network to process inertial data to obtain the motion
features from a sequence of inertial data. It combines the visual features and
inertial features to form the fused features. [15] adds an attention module
into the visual-inertial odometry network to improve the robustness, and shows
better performance in multiple complex environments. [16] introduces to fuse
point clouds with inertial information to form a multi-sensor odometry
estimation network. [17] introduces a memory and refining module that preseves
important contextual information and based on that further improves poses.
DAVO [18] improves the performance of supervised learning based VO by fusing
the inputs from semantic segmentation, optical flow and RGB image via an
attention module.

###  II-C The Unsupervised learning of ego-estimation estimation

Unsupervised learning of egomotion methods have attracted attention, as they
do not require high-precision pose labels. SfMLearner[19] designs a double
network structure: PoseNet and DepthNet. It takes three consecutive images as
the optimization sequence. PoseNet is responsible for estimating the relative
pose between each two images. DepthNet is responsible for predicting the
target frame depth. It combines relative pose estimates, target depth
prediction and target/source images pixels to construct the photometric loss.
In order to improve the accuracy of depth prediction, DepthNet adopts
U-Net[20] like structure, which connects the multi-scale feature map of
decoder and encoder. [21] uses ResNet18[22] as the backbone of PoseNet and
DepthNet. In order to obtain a more accurate pose and depth estimate, it
upsamples different resolution depth predictions from DepthNet decoder to raw
image resolution. And then it combines multi-scale depth predictions to form
photometric loss. Besides, it adds the per-pixel minimum reprojection loss to
eliminate the outliers in the photometric loss. [23] predicts disparity map at
all scales in DepthNet encoder and decoder, and it transforms disparity maps
into pseudo-labels. It uses pseudo-labels as self-distillation signals in
training process. Meanwhile, it introduces depth hint labels into the
photometric loss to facilitate depth learning. To avoid overfitting, it also
proposes a data augmentation method for depth estimation. In order to solve
the problem of inconsistent depth prediction scales, [24] proposes a depth
scale consistency loss to constrain the depth and pose estimate to a unified
scale. After multiple training iterations, the depth prediction in whole
training set will keep a global consistent scale metric. [25] designs a
separate scale-net to predict the scale of each depth map. In order to obtain
the pose and depth with absolute scale, PackNet-SfM[26] introduces
instantaneous velocity to calculate the adjacent frames displacement as weak
velocity supervision. PackNet-SfM combines weak velocity supervision with the
relative displacement estimate from PoseNet to construct velocity supervision
loss. In order to introduce IMU data into unsupervised work, VIOlearner[1]
used CNN to process IMU data and designed online error correction module to
calculate multi-scale photometric errors. They interpolate the sparse point
clouds to ground truth depth labels, and then they integrate the depth labels
in photometric error with adjacent images. The performance of odometry is
better than that of end-to-end VINet, but this work does not include depth
estimation, so it can not reconstruct the scene in whole trajectory. Self-
VIO[27] transforms IMU data into inertial features through CNN and combines
them with visual features to obtain fusion features. A fusion selection module
is built to filter fusion features, and a GAN[28] discriminator is designed to
optimize the depth prediction to obtain high-frequency structure. At the same
time, LSTM network is also used to comprehensively optimize the multi frame
pose and depth predictions, Experimental results show that the performance of
the proposed VIO is comparable to that of the mainstream model-based VIO.
UnVIO[29] introduces two optimization modules: intra-window optimization and
inter-window optimization. It constructs the photometric error between two
adjacent frames and smooth loss in intra window optimization. In inter window
optimization, integrated information and pose prediction of odometry are used
to construct trajectory consistency loss and 3D geometric consistency loss.
Two kinds of consistency loss can constrain the global pose scale, and make
pose prediction more consistent. However, the network cannot obtain pose
results with absolute scale.

##  III Self-supervised Visual Egomotion and Depth Learning

###  III-A Framework

The overview framework of our proposed self-supervised ego-motion and depth
learning is shown in Figure 1. Inspired by [19], our framework also adopt
double network design: PoseNet and DepthNet. Our PoseNet includes visual
encoder, inertial encoder, attention based feature fusion module and pose
estimation module. The framework inputs a sequence of images, and outputs the
corresponding poses and depth maps. A pair of adjacent images are feed into
ResNet18 based visual feature encoder to extract multi-state visual features,
after which a LSTM network is used to model their temporal dependency. The
visual features after LSTM are input into fully connected layers (FC layers)
to produce the relative pose between the target frame and the source frame.
DepthNet estimates the depth of each image in the image sequence. In addition,
inertial data can be introduced into the framework to formulate visual-
inertial PoseNet. Inertial feature extractor encodes inertial features from a
sequence of IMUs. Then, an attention module is used to combine visual features
and inertial features, which are further feed into LSTM. Our framework is
trained in two steps: 1.Coarse Scale Recovery, it makes the PoseNet output
pose with coarse absolute scale. 2. Coarse To Fine Scale Recovery, it refines
the PoseNet pose prediction and updates the DepthNet parameters.

####  III-A1 Visual Encoder

We use ResNet18 backbone as the visual feature encoder $f_{\text{visual}}$.
Two adjacent images $\left\\{\mathbf{I}_{i},\mathbf{I}_{i+1}\right\\}$ are
combined on the RGB channel to obtain 6-dimensional tensors, which are
inputted into the visual feature encoder to obtain visual features in 1/4
input image size. In order to simplify the network structure and improve the
forward efficiency, we use global average pooling to transform ith frame
visual features into a 128 dimensional feature vector $\mathbf{v}_{i}$.

$\mathbf{v}_{i}=f_{\text{visual}}\left\\{\mathbf{I}_{i},\mathbf{I}_{i+1}\right\\}$
(1)

####  III-A2 Inertial Encoder

The data acquisition frequency of IMU is higher than that of camera, and it
has high measurement accuracy in a short time. Between two adjacent images
$\left\\{\mathbf{I}_{i},\mathbf{I}_{i+1}\right\\}$, there exist j imu raw data
${M}^{0\cdots{j-1}}_{i\to{i+1}}$. The jth imu frame includes linear
acceleration $\alpha_{j}$ and angular velocity $\omega_{j}$ in body
coordinate, is one-dimensional time series data, so we introduce the Bi-
directional LSTM network as inertial feature encoder $f_{\text{inertial}}$,
which excels at processing time series data. The input dimension of two-layer
Bi-directional LSTM network is 6, the number of hidden state
$\left\\{\mathcal{H}_{j-1},\mathcal{H}_{j}\right\\}$is 128, and the dimension
of jth frame output inertial feature vector $\mathbf{a}_{j}$ is 256.

$\mathbf{M}_{i\to{i+1}}=\begin{bmatrix}\alpha^{0}_{i}&\omega^{0}_{i}\\\
\cdots&\cdots\\\
\alpha^{j-1}_{i+1}&\omega^{j-1}_{i+1}\end{bmatrix}\in\mathbb{R}^{j\times 6}$
(2)

$\mathbf{a}_{i},\mathcal{H}_{i}=f_{\text{inertial}}\left\\{({\alpha_{i}},{\omega_{i}});\mathcal{H}_{i-1}\right\\}$
(3)

####  III-A3 Attention based Feature Fusion Module

There may be a few misalignment in the raw data, bias in the IMU data, and
errors in the calibration of intrinsics and extrinsics parameters. If the
features of the two kind of data are directly combined, the odometry
performance is likely to be suboptimal. Even in the pure visual feature, image
features with missing texture will weaken the performance of pose estimation.
Inspired by [15], we introduce the attention module in PoseNet. Therefore,
poseNet can focus more on features that are conducive to improving the
performance of pose estimation. We concatenate two kind of features
$\left\\{\mathbf{a_{i}};\mathbf{v_{i}}\right\\}$ to form the concatenated
features. And then we use the attention mask [15] to filter the concatenated
features. In the end we obtain the filtered fusion features $\mathbf{z_{i}}$.

$\mathbf{z_{i}}=f_{\text{attention}}\left\\{\mathbf{a_{i}};\mathbf{v_{i}}\right\\}$
(4)

####  III-A4 Pose Estimation Module

The filtered fusion features are brought into pose estimation module. In this
module, there exist a LSTM network and a FC layer. The LSTM network integrates
the filtered multi fusion features $\mathbf{z_{i,i\in{0\cdots n-1}}}$ in one
sequence window, sequence length is $n$. We set the hidden layer nodes in LSTM
network to 1024 and the high-level output features $\mathbf{\bar{z}_{i}}$
dimension to 2048. After the LSTM network, we use a dropout layer with a
coefficient of 0.2 to further prevent the possibility of over-fitting. We
subsequently send the high-level output features to the FC layer to obtain
6-DOF relative poses $\mathbf{T^{i+1}_{i}}$.

$\mathbf{\bar{z}_{i}}=\text{LSTM}\left\\{\mathbf{{z}_{i}}\right\\}_{i\in
0\cdots{n-1}}$ (5)

$\mathbf{T^{i+1}_{i}}=\text{FC}\left\\{\mathbf{\bar{z}_{i}}\right\\}_{i\in
0\cdots{n-1}}$ (6)

####  III-A5 DepthNet

We use U-Net like DepthNet, which is divided into encoder and decoder. The
encoder of the DepthNet adopts the ResNet18 backbone, which is pretrained on
ImageNet data set. DepthNet encoder can extract the visual feature in input
image. The decoder of the DepthNet combines the feature maps from multi-scale
encoder layers. There exist multiple convolution layers and nearest neighbor
interpolations in DepthNet decoder. Therefore, DepthNet decoder can upsample
the visual feature to original image resolution. We take the DepthNet decoder
output as the disparity map. In the end we use the reciprocal of the disparity
map as the depth prediction.

Figure 2: coarse depth scale recovery

####  III-A6 Photometric loss

The pixel points of the same 3D projection in two adjacent frames have
brightness constancy and spatial smoothness. Given the estimated relative pose
between the target frame $\mathbf{I}_{t}$ and the source frame
$\mathbf{I}_{s}$, the depth of target frame $\mathbf{D}_{t}$ and camera
intrinsic $\mathbf{K}$, we can generate the synthesized image
$\mathbf{\hat{I}}_{s}$ by warping $\mathbf{I}_{s}$ into target-view:

$\mathbf{p}_{s}\sim\mathbf{K}\mathbf{T}_{t}^{s}\mathbf{D}_{t}(\mathbf{p}_{t})\mathbf{K}^{-1}\mathbf{p}_{t},$
(7)

where $\mathbf{p}_{s}$ and $\mathbf{p}_{t}$ indicate pixel point in source
frame and target frame.

This work aims to recover the scale of ego-motion and depth learning. If the
odometry pose scale is not recovered, it is difficult for odometry to be
applied in realistic scenes. The reasons why the monocular unsupervised
odometry cannot recover the pose scale: 1. The monocular image cannot
determine the absolute distance of the 3D point in the single image. 2. It is
impossible to determine the scale of $\mathbf{T}_{t}^{s}$’s translation vector
and $\mathbf{D}_{t}$ only with photometric loss as a constraint.

###  III-B Coarse Depth Scale Recovery via Photometric-loss-consistent
Pretrained Model

We firstly introduce a depth scale recovery method via photometric-loss-
consistent pretrained model. Unlike previous works[1][30], we do not use dense
depth completion results as supervisions. This is due to the fact that the
performance of depth completion is closely related to the point clouds
density. If point clouds are too sparse, its performance will be degraded. The
dense depth completion methods can be divided into interpolation-based and
learning-based methods. [1] completes the sparse point clouds by
interpolation. However, the interpolated areas do not necessarily conform to
the multi-view geometry, and thus the generated depths are not suitable to
being applied in learning depth recovery. The learning-based methods [30, 31]
learn to generate dense depths from sparse depths, which can mitigate such
problems to some extent, but there still exist completion areas that do not
conform to multi-view geometry, such as out of the LIDAR scanning range. And
it is too complicated to remove outliers before introducing these completion-
based depths as supervisions into photometric loss for scale recovery.

To handle the problem mentioned above, in our framework, we propose to use a
pre-trained DepthNet to produce the depth predictions, and the scale of depth
prediction is recovered by comparing them with the corresponding point clouds.
The depth predictions from pre-trained model naturally conform to the multi-
view geometry. The depth predictions from DepthNet are more smooth and
complete, and there is no need to remove abnormal values in advance. We use
the scale-recovered depth prediction as the supervisions for the recovery of
pose and depth estimates.

We firstly transform the point cloud vector $\mathbf{v}_{4\times 1}$ from
LIDAR to the pixel coordinate as the depth projection point $\mathbf{d}_{v}$.

$\mathbf{d}_{v}=\mathbf{P}_{c2}\mathbf{T}^{\text{rect}}_{c0}\mathbf{T}^{c0}_{v}\mathbf{v}_{4\times
1}$ (8)

where $\mathbf{P}_{c2}$ is projection matrix, $\mathbf{T}^{\text{rect}}_{c0}$
is transformation matrix from camera $c0$ coordination to rectified camera
$c0$ coordination, $\mathbf{T}^{c0}_{v}$ is transformation matrix from lidar
$\mathbf{v}$ coordination to camera $c0$ coordination. We form the depth
projection image $\mathbf{D}_{v}$ from depth projection
points$\mathbf{d}_{v}$.

Figure 3: The calculation of forward loss and backward loss

Then we calculate the ratio between the median/mean value of depth projection
images $\text{med}(\mathbf{D}_{v})$ and the median/mean value of depth
predictions from pre-trained DepthNet $\text{med}(\mathbf{D}_{t})$. The scale
factor $\varepsilon$ is calculated as below,

$\varepsilon=\frac{\text{med}(\mathbf{D}_{v})}{\text{med}(\mathbf{D}_{t})}$
(9)

Finally, the scale factor $\varepsilon$ is multiplied with the depth
predictions to obtain the depths with absolute scale $\bar{\mathbf{D}}$.

$\bar{\mathbf{D}}=\varepsilon\cdot\mathbf{D}$ (10)

###  III-C Single-directional Coarse Pose Scale Recovery

The depths with absolute scale are further leveraged to recover the pose scale
of our self-supervised learning framework. We propose a two-stages process to
achieve this. The first stage aims to estimate coarse poses by exploiting the
calculated coarse depths, and then the second stage refines the network
estimates by jointly optimizing poses and depths.

Firstly, coarse poses are obtained by integrating the calculated coarse depths
from Equation 10 into the Equation 7 to produce the synthesized images
$\hat{\mathbf{I}}_{s}$ in target frame. We calculate the difference between
the wrapped images $\hat{\mathbf{I}}_{s}$ and the real image in target frame
$\mathbf{I}_{t}$ to construct a photometric loss. Besides, SSIM loss is added
to mitigate complex illumination changes [32]. We formulate our photometric
loss $L_{\text{photo}}$ as below,

$\mathcal{L}_{\text{pho}}=\lambda_{1}\cdot|\mathbf{I}_{t}-\mathbf{\hat{I}}_{s}|+\lambda_{2}\cdot\frac{1-\text{SSIM}(\mathbf{I}_{t},\mathbf{\hat{I}}_{s})}{2},$
(11)

where $\lambda_{1}$ and $\lambda_{2}$ are set as 0.15 and 0.85. The specific
function of SSIM can refer to [32]. (please add SSIM function)

To encourage pose and depth predictions to be scale consistent, a 3D geometric
consistency loss $L_{\text{GC}}$[24] is introduced:

$\mathcal{L}_{\text{GC}}=\frac{|\mathbf{\hat{D}}_{s}-\mathbf{T}_{t}^{s}\mathbf{D}_{t}|}{\mathbf{\hat{D}}_{s}+\mathbf{T}_{t}^{s}\mathbf{D}_{t}},$
(12)

where $\mathbf{T}_{t}^{s}\mathbf{D}_{t}$ indicates the process of wrapping the
target depth $\mathbf{D}_{t}$ into the source frame view using the relative
pose $\mathbf{T}_{t}^{s}$. And $\mathbf{\hat{D}}_{s}$ is the interpolated
source depth map aligning with $\mathbf{T}_{t}^{s}\mathbf{D}_{t}$.

Most existing unsupervised ego-motion networks optimize the learned pose and
depth between adjacent frames. To consider the temporal dependency of poses
and depths in consecutive frames, our framework optimizes them in a sequence
window, which further encourages the pose scale-recovery to be more smooth in
a larger window size. We set $i$ as the time step of the target frame, $i+1$
as the time step of source frame, and $n$ as the sequence window length. In
each time step, we combine photometric loss $\mathcal{L}^{i}_{\text{pho}}$ and
3D geometric consistency loss $\mathcal{L}^{i}_{\text{GC}}$ to form an
optimization loss. Combining the total losses of $n$ frames, we form the
single direction scale-recovery loss $\mathcal{L}^{i\to i+1,i\in n}$ as below,

$\mathcal{L}^{i\to
i+1}_{\text{forward}}=\lambda_{3}\cdot\sum_{i=1}^{n}{\mathcal{L}^{i}_{\text{pho}}}+\lambda_{4}\cdot\sum_{i=1}^{n}{\mathcal{L}^{i}_{\text{GC}}}$
(13)

where $\lambda_{3}$ and $\lambda_{4}$ are set as 1 and 0.5.

We use the calculated coarse depths
$(\bar{\mathbf{D}}_{t},\bar{\mathbf{D}}_{s})$ as the target and source depth
in this stage to construct a coarse scale recovery loss
$\mathcal{L}_{\text{coarse}}$,

$\mathcal{L}_{\text{coarse}}=\mathcal{L}^{i\to
i+1}_{\text{forward}}|_{(\bar{\mathbf{D}}_{t},\bar{\mathbf{D}}_{s})}$ (14)

The model is trained for several epochs to obtain a coarse scale recovered
PoseNet, and then the entire framework is further refined in the next stage
for jointly optimizing PoseNet and DepthNet together.

###  III-D Bi-directional Coarse-to-Fine Pose Scale Recovery

In the coarse scale recovery stage, the depths are calculated from pre-trained
model and estimated scale factor. This stage considers to enable pose and
depth learning jointly, which further refines pose and depth estimates.

Instead of using calculated depths, we use the depth predictions from DepthNet
as the target and source depths to construct the photometric loss and 3D
geometry consistency loss of coarse-to-fine scale recovery. If the
optimization is only on the forward-direction, the model encourages depth
predictions to be overfit on one side, leading to unstable training.
Therefore, we introduce a backward loss,

$\mathcal{L}^{i+1\to
i}_{\text{back}}=\lambda_{3}\cdot\sum_{i=1}^{n}{\mathcal{L}^{i}_{\text{pho}}}+\lambda_{4}\cdot\sum_{i=1}^{n}{\mathcal{L}^{i}_{\text{GC}}}$
(15)

Here, as shown in Figure 3, the relative pose and the order of target and
source frame are inverse.

Furthermore, we introduce an edge-aware smoothness loss
$\mathcal{L}_{\text{smooth}}$ to overcome the shortage of photometric loss in
the low-texture regions [33].

$\mathcal{L}_{\text{smooth}}=\sum_{p\in(u,v)}(e^{-\bigtriangledown\mathbf{I}_{t}(p)}\cdot\mathbf{D}_{t}(p))^{2}$
(16)

where $p$ indicates pixel point in coordination $(u,v)$, and
$\mathbf{I}_{t}(p)$ indicates the pixel value in target frame
$\mathbf{I}_{t}$, $\bigtriangledown$ means the first derivative along two
spatial directions on pixel coordination $(u,v)$.

Combining the forward $\mathcal{L}^{i\to i+1}$ loss, backward
$\mathcal{L}^{i+1\to i}$ loss and smoothness loss
$\mathcal{L}_{\text{smooth}}$, the total loss for bi-directional scale
recovery is formulated as:

$\mathcal{L}_{\text{refine}}=\lambda_{5}(\mathcal{L}^{i\to
i+1}_{\text{forward}}+\mathcal{L}^{i+1\to
i}_{\text{back}})|_{({\mathbf{D}_{t}},{\mathbf{D}_{s}})}+\lambda_{6}\cdot\mathcal{L}_{\text{smooth}}$
(17)

where we set $\lambda_{5}=1$ and $\lambda_{6}=0.1$.
${({\mathbf{D}_{t}},{\mathbf{D}_{s}})}$ are the learned depth from DepthNet.

###  III-E Discussion

We have also tried to use coarse scaled depth predictions to directly train
the overall framework, instead of dividing training process into two stages:
coarse scale recovery and coarse to fine scale recovery. However, the
experimental results show that the pose and depth scale cannot be recovered
directly by introducing depth labels. We analyze that since the absolute-scale
depth supervisions are non-learnable, at the beginning of training, the
photometric loss formed by the depth supervision is large, and the loss
reduction potential is small. The photometric loss formed by the learnable
depth prediction will decrease with the training iteration, and the loss
reduction potential is large. Besides, the direction of network training is
consistent with the direction of loss reduction. Therefore, the scaled depth
supervision can not help framework to recover absolute scale in direct
training.

##  IV Experiment

This section introduces implementation details, and discusses the evaluation
of pose and depth estimation of our proposed model above two datasets. In
addition, ablation study is conducted to verify the effectiveness of the
important modules in our model.

TABLE I: The pose performance on the KITTI dataset, ”Scale” indicates whether the pose estimates are scaled or not. Method | Sensors | Scale | Seq. 09 | Seq. 10 | Avg
---|---|---|---|---|---
$t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$
ORB-SLAM | Mono | x | 15.3 | 0.26 | 3.68 | 0.48 | 9.49 | 0.37
Depth-VO-Feat | Stereo | x | 11.92 | 3.60 | 12.62 | 3.43 | 12.27 | 3.515
GeoNet | Mono | x | 23.94 | 9.81 | 20.73 | 9.10 | 22.34 | 9.46
Monodepth2 | Mono | x | 18.12 | 3.86 | 12 | 5.34 | 15.06 | 4.6
SfMLearner | Mono | x | 17.84 | 6.78 | 37.91 | 17.78 | 27.875 | 12.28
SC | Mono | x | 8.62 | 3.05 | 7.81 | 4.9 | 8.22 | 3.98
Ours(VO) | Mono | ✓ | 7.90 | 1.23 | 6.55 | 2.05 | 7.23 | 1.64
VINS | Mono+IMU | ✓ | 41.47 | 2.41 | 20.35 | 2.73 | 30.91 | 2.57
VINet | Mono+IMU | ✓ | 11.83 | 3.00 | 8.60 | 4.39 | 10.22 | 3.70
UnVIO | Mono+IMU | x | 4.41 | 0.92 | 6.42 | 0.63 | 5.41 | 1.55
Ours(VIO) | Mono+IMU | ✓ | 5.48 | 0.19 | 5.37 | 0.43 | 5.43 | 0.31
Ours(VIO) (Scaled) | Mono+IMU | ✓ | 3.65 | 0.19 | 4.42 | 0.43 | 4.04 | 0.31

###  IV-A Training details

We use PyTorch to implement our proposed network. Our model is trained and
tested via a NVIDIA RTX3090 GPU. Adam is chosen as optimizer to recover the
optimal parameters, whose attenuation coefficient is $\beta_{1}=0.9$,
$\beta_{2}=0.999$. The model is trained for 200 epochs, and in each epoch
there are 1000 random data sequences. The model training follows a two-stages
process: first, we use the pseudo depth labels to train the network for around
30-100 epochs, until a coarse pose estimation with absolute scale can be
recovered; then, depth and pose networks are jointly trained to enable coarse-
to-fine pose and depth estimation. The learning rate is chose as 1e-4 in the
first stage and reduced to 1e-5 in the second stage.

###  IV-B Datasets

####  IV-B1 KITTI Odometry Dataset

The KITTI Raw dataset is a typical car-driving dataset. It provides RGB
images, IMU, LIDAR point cloud data and ground-truth pose data. We use this
dataset to generate pseudo depth labels and odometry network training data.
The KITTI Raw dataset for odometry includes 10 sequences. We use Sequence
00-08 excluding Sequence 03 for training and evaluation, as the IMU data of
Sequence 03 are missing, while Sequence 09 and 10 are used for testing. Since
the IMU and LIDAR data in the KITTI Raw dataset are unsynchronized, we
manually synchronize these data according to their timestamps. The frequency
of IMU is 100Hz, while the frequency of images and the synchronized LIDAR
point cloud data is 10Hz. The ground-truth pose is not involved in training
process, but used in the testing process to evaluate the pose accuracy.

####  IV-B2 MVSEC Dataset

In order to evaluate our model performance at night, we select the Multi
Vehicle Stereo Event Camera (MVSEC) dataset, which collected gray images and
IMU data. The images were sampled at a frequency of 20Hz and IMU data were
recorded at a frequency of 200Hz. In our experiments, we use the data from
several car-driving scenarios in daytime and at night, including two day-time
sequences (Day1, Day2) and three night-time sequences (Evening1, Evening2,
Evening3). We used the Day1, Day2, Evening1, and Evening2 as training data,
and the Evening3 as testing data. From the figure 4, it can be seen that the
lighting conditions at night impose challenges to the visual pose and depth
estimation. The dataset also provides ground-truth pose, which is only used in
the testing process for evaluation.

(a) Day-time image

(b) Night-time image

Figure 4: The sample images of the MVSEC dataset: a) driving in daytime b)
driving at night

(a) KITTI Sequence 09

(b) KITTI Sequence 10

Figure 5: The trajectories of our proposed self-learning based VO and VIO
model on Sequence 09 and 10 of the KITTI dataset, comparing with baselines.

###  IV-C Pose evaluation on the KITTI dataset

We first come to evaluate our proposed VO/VIO models on the public-available
KITTI dataset. Following the previous research, we use Sequence 00-08 for
training and evaluation, and Sequence 09 and 10 for testing. The KITTI’s
official evaluation is adopted here. It is to calculate the RMSE of
translation vector and rotation of the sequences with a length from 100m to
800m, and average them as the criterion of pose accuracy.

TABLE II: The depth evaluation on the KITTI dataset. These models are trained with Sequence 00-08, and tested with Sequence 09 and 10. Method | Sensors | Resolution | Scale | Error metric | Accuracy metric($\delta$ )
---|---|---|---|---|---
Abs Rel $\downarrow$ | Sq Rel $\downarrow$ | RMSE $\downarrow$ | (1.25) $\uparrow$ | (1.25^2) $\uparrow$ | (1.25^3) $\uparrow$
SfMLearner | Mono | $832\times 256$ | x | 0.3272 | 3.1131 | 9.5216 | 0.4232 | 0.7010 | 0.8476
SC | Mono | $832\times 256$ | x | 0.1629 | 0.9644 | 4.9129 | 0.7760 | 0.9315 | 0.9773
UnVIO | Mono+IMU | $832\times 256$ | x | 0.1322 | 0.73005 | 4.2443 | 0.8324 | 0.9509 | 0.9821
Ours1 | Mono | $832\times 256$ | x | 0.1322 | 0.7244 | 4.2715 | 0.8313 | 0.9498 | 0.9816
Ours2 | Mono | $832\times 256$ | ✓ | 0.1308 | 0.6980 | 4.2930 | 0.8318 | 0.9520 | 0.9836
Ours3 | Mono+IMU | $832\times 256$ | ✓ | 0.1301 | 0.6942 | 4.2963 | 0.8341 | 0.9526 | 0.9837

Input SfM SC Ours1 Ours2

Figure 6: The samples of depth estimation from SfM-Learner, SC and our
proposed models

For vision only pose estimation, we compare with a mainstream traditional SLAM
(i.e. ORB-SLAM) and five learning based VOs (i.e. Depth-VO-Feat, GeoNet,
MonoDepth2, SfMLearner, SC). Among the learning based baselines, Depth-VO,
GeoNet, MonoDepth2, SfMLearner are scale-ambiguity, while SC is scale-
consistent, but still loses its absolute scale. However, our proposed self-
supervised VO can produce pose with global scale. From Table I, it is clear to
see that our model outperforms all both traditional and learning based
baselines in translation, and outperforms all the learning based VOs in
rotation. Our VO model further improves SC 12.04% in translation and 58.79% in
rotation averagely. Though the rotation estimation of ORB-SLAM is still better
than our VO model slightly, we show that by introducing inertial data into our
model as learning based VIO, our VIO model shows superior performance in
rotation than all the VO/VIO baselines.

Our VIO model is compared with a representative traditional VIO, i.e. VINS,
and two learning based VIO, i.e. VINet and UnVIO. VINet is a supervised
learning based VIO model using a combination of ConvNet and LSTM. UnVIO is an
unsupervised learning based VIO model via novel view synthesis similar to our
model, but it has no absolute scale metric. Clearly, our model significantly
outperforms these baselines in both translation and rotation. Our VIO model
improves the translation and rotation estimation of UnVIO around 25.32% and
80% respectively. As illustrated in Figure 5, the generated trajectories from
our VO and VIO models are closer to the ground-truth trajectories, compared
with other baselines, i.e. VINS-mono, MonoDepth2, SC, ORB-SLAM and VINet.

###  IV-D Depth evaluation on the KITTI Dataset

In addition to pose estimation, we also evaluate the depth evaluation on the
KITTI dataset. Different from the SC-SfMLearner that use KITTI raw dataset for
training, we use Sequence 00-08 of KITTI as these sequences are with IMU data.
Table II shows the quantitative results of the depth estimation from our
proposed models on the Sequence 09 and 10, comparing with three mainstream
learning based depth estimation methods, i.e. SfMLearner, SC and UnVIO.

In Table II, we use following evaluation metrics for depth estimation, where
$p$ indicates pixel point in coordination $(u,v)$, $\mathbf{D}$ and
$\mathbf{D}_{v}$ indicate learned depth prediction from DepthNet and
corresponding depth projection image. $a$ is the threshold factor. $N$ is the
sum of pixel points in the depth image coordinate.

Abs Rel
$\displaystyle=\frac{1}{N}\sum_{p}\frac{\left|\mathbf{D}(p)-\mathbf{D}_{v}(p)\right|}{\mathbf{D}_{v}(p)}$
(18) Sq Rel
$\displaystyle=\frac{1}{N}\sum_{p}\frac{(\mathbf{D}(p)-\mathbf{D}_{v}(p))^{2}}{\mathbf{D}_{v}(p)}$
RMSE
$\displaystyle=\sqrt{\frac{1}{N}\sum_{p}(\mathbf{D}(p)-\mathbf{D}_{v}(p))}$
$\displaystyle\delta(a)$
$\displaystyle=\frac{1}{N}\sum_{p}{\text{max}(\frac{\mathbf{D}_{v}(p)}{\mathbf{D}(p)},\frac{\mathbf{D}_{(}p)}{\mathbf{D}_{v}(p)})<a}$

It indicates that our depth estimation models (Ours2 and Ours3) can produce
depth in absolute scale and outperform other three baselines. The introduction
of IMU into depth estimation throws little influence upon depth estimation.
Figure 6 shows several depth images from monocular camera, generated by our
proposed models and baselines. We can see that compared with other learning
based depth estimation models, our models can produce depth maps with more
delicate details, e.g., the poles can be clearly shown in our depth maps.

TABLE III: A comparison of the scale factor estimation with or without our introduce scale recovery method. Method | Scale | $\mu$ | $\delta$
---|---|---|---
Ours | x | 33.109 | 4.384
Ours | ✓ | 1.128 | 0.118

Moreover, we continue to study the effectiveness of our introduced scale
recovery method. Table III shows the mean value and standard deviation value
by comparing the scale factor of depth estimation with or without scale
recovery. It can be seen that the averaged scale coefficient of depth
estimation values is closer to 1 when adopted our proposed coarse-to-fine
scale recovery method. If scale recovery is not performed, there is a large
distance between the estimated scale-ambiguous depths and real depths. This
indicates that the introduction of scale recovery can not only enable PoseNet
to output pose predictions with global absolute scale, but also help DepthNet
output depth estimates closer to the real absolute scale metric.

###  IV-E Pose and Depth Evaluation on the Night Scenes of MVSEC Dataset

To evaluate our proposed model on very challenging night scenes, which are in
low-light conditions and relatively featureless, we selected the MVSEC dataset
to conduct experiments. A recent state-of-the-art monocular SLAM, (i.e. ORB-
SLAM3), a representative unsupervised-learning based VO (i.e. SC-SfMLearner)
and a supervised learning based (i.e. DeepVO) are chosen for a comparison.

The trajectories of each model on Evening3 are displayed in Figure 7. It can
be seen seen that the learning based VOs can robustly estimate vehicle ego-
motion and generate trajectories in this challenging scene. However, the
traditional model, ORB-SLAM3-Mono fails. This might be due to fact that ORB-
SLAM3 Mono can extract and match enough feature points under low-light
environment, and thus are not capable of performing effective feature tracking
for pose estimation, leading to positioning failure. In contrast, deep
learning networks excel at learning and extracting features and thus enable
learning-based methods to extract sufficient and reliable visual feature for
positioning.

We use RMSE as the evaluation metric via Equation 19, and further compare
these quantitative results in Table IV,

$\text{Error}=\sqrt{\frac{1}{n}(\mathbf{x}_{i}-\mathbf{x}^{\text{gt}}_{i})}$
(19)

Where $\mathbf{x}_{i}$ represents the estimated three-dimensional location in
$ith$ frame, and $\mathbf{x}^{\text{gt}}_{i}$ is the corresponding ground-
truth location. The quantitative results show that the performance of our
proposed VO model exceeds both unsupervised learning based VO, i.e. SC, and
supervised learning based VO, i.e. DeepVO. It demonstrates that our framework
can robustly and accurately estimate pose in low light environment.

TABLE IV: The pose estimation results in the night car-driving scenes. Method | Sensors | Scale | Error
---|---|---|---
ORB-SLAM3 Mono | Mono | ✓ | failed
SC-SfMLearner | Mono | x | 72.05
DeepVO | Mono | ✓ | 39.37
Ours(VO) | Mono | ✓ | 35.12
Ours(VIO) | Mono+IMU | ✓ | 19.62
Figure 7: The generated trajectories in the night car-driving scenes

In addition, we qualitatively evaluate the depth estimation performance of our
framework in the challenging night scene, comparing with SC. Figure 8 shows
input RGB image, depth maps from LIDAR data, depth estimation from SC, and
depth estimation from our our model. Clearly, SC does not perform well when it
is in the area of strong light. SC will misleadingly estimates depth in a
shorter distance in strong-light condition. It might be because our model
introduces a coarse-to-fine stage into depth estimation that enables DepthNet
to overcome the influence of light condition, e.g. too strong or too dark.

Input GT SC Ours

Figure 8: The depth predictions from our model and baselines in low-light
condition of driving at night.

###  IV-F Ablation Study

Finally, we conduct ablation study to verify the effectiveness of different
modules in our framework. We use the KITTI dataset and learning based VIO
model. The results of ablation study into framework modules are shown in Table
V. In order to test the impact of the attention module on pose estimation, we
remove attention mask and directly input the fusion features of vision and
inertial into the subsequent LSTM and PoseNet. We can see that the attention
module can further improve the performance of pose network, especially
rotation estimation. Also, the addition of IMU data into framework can enhance
the network performance. This indicates the effectiveness of multimodal
fusion, as inertial data contributes to rotation estimation more, and the
fusion of features can compensate the shortcomings of different modalities. In
order to evaluate the effectiveness of LSTM network, we remove LSTM module,
and input the fused features directly into PoseNet, showing that adding LSTM
module enhances model performance.

TABLE V: The ablation study into framework modules Method | IMU | LSTM | Attention | Seq.09 | Seq.10 | Avg
---|---|---|---|---|---|---
$t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$
Ours |  | $\checkmark$ | $\checkmark$ | 8.84 | 1.31 | 7.34 | 1.98 | 8.09 | 1.64
Ours | $\checkmark$ |  | $\checkmark$ | 14.57 | 0.56 | 10.92 | 1.73 | 12.74 | 1.15
Ours | $\checkmark$ | $\checkmark$ |  | 5.06 | 0.29 | 6.33 | 0.57 | 5.70 | 0.43
Ours | $\checkmark$ | $\checkmark$ | $\checkmark$ | 5.48 | 0.19 | 5.37 | 0.43 | 5.43 | 0.31

Then, we study the impact of different sequence lengths on pose estimation,
and the comparison results are shown in Table VI. Compared with a short
sequence length (3 frames), we can see that a longer window length (a sequence
of 5 frames) can improve both translation and rotation estimation accuracy,
especially the rotation improve is more significant.

TABLE VI: The ablation study into sequence length Method | Window length | Seq.09 | Seq.10 | Avg
---|---|---|---|---
$t_{\text{rel}}$ | $r_{\text{rel}}$ | $t_{\text{rel}}$ | $r_{\text{rel}}$ | $t_{\text{rel}}$ | $r_{\text{rel}}$
Ours1 | 3 | 6.42 | 0.45 | 4.90 | 0.55 | 5.66 | 0.5
Ours2 | 5 | 5.48 | 0.19 | 5.37 | 0.43 | 5.43 | 0.31

Finally, the influence of adding scaled depth predictions and learned depth on
pose estimation is studied. The comparison results are shown in Table VII. In
order to verify the effectiveness of the learnable DepthNet on pose
estimation, Ours1 follows coarse-to-fine scale recovery training, but it uses
the scaled depth predictions instead of the learned depths from DepthNet, to
form the total loss for bi-directional scale recovery, and the learning rate
is consistent with Ours3. Although the scaled depth predictions must be higher
in accuracy than the learned depths from DepthNet, the scaled depth
predictions are unlearnable and cannot be optimized with the iteration of
training. The potential for scaled pseudo depth labels to improve the pose
estimation accuracy is thus limited. In order to verify the effectiveness of
the learned depth from DepthNet on pose estimation, Ours2 does not introduce
any supervision in framework, but directly uses learned depth to form the
total loss for bi-directional scale recovery, and the learning rate is 1e-4.
It can be seen that the performance Ours3 with our proposed training strategy
is higher than Ours1 and Ours2.

TABLE VII: The ablation into supervision signal and learning rate, ”C” indicates Coarse Scale Recovery, ”F” indicates Coarse-to-fine Scale Recovery, ”D” indicates scaled depth predictions, ”-” means this stage does not exist, ”LR” indicates learning rate. Method | Supervision | LR | Seq.09 | Seq.10 | Avg
---|---|---|---|---|---
C | F | C | F | $t_{\text{rel}}$ | $r_{\text{rel}}$ | $t_{\text{rel}}$ | $r_{\text{rel}}$ | $t_{\text{rel}}$ | $r_{\text{rel}}$
Ours1 | D | D | 1e-4 | 1e-5 | 8.01 | 1.07 | 9.06 | 1.45 | 8.54 | 1.26
Ours2 | - | x | $\times$ | 1e-4 | 6.06 | 0.66 | 7.65 | 2.25 | 6.86 | 1.45
Ours3 | D | x | 1e-4 | 1e-5 | 5.48 | 0.19 | 5.37 | 0.43 | 5.43 | 0.31

##  V Conclusion

In this work, we propose a self-supervised learning based depth and egomotion
estimation framework with a novel coarse-to-fine scale recovery strategy. Our
model can accurately and robustly output pose and depth with global scale
metric, even in low-light conditions at night. We conducted extensive
experiments to evaluate our model on two public datasets, i.e. KITTI and
MVSEC. On both datasets, our model outperforms both traditional and learning
based VOs/VIOs. Ablation study is also performed to verify the effectiveness
of each module in our framework. In the future, we will attempt to exploit IMU
data for scale recovery and pose estimation that further contributes to the
performance improvement.

## References

  * [1] E. J. Shamwell, K. Lindgren, S. Leung, and W. D. Nothwang, “Unsupervised deep visual-inertial odometry with online error correction for rgb-d imagery,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 42, no. 10, pp. 2478–2493, 2019.
  * [2] B. Kitt, A. Geiger, and H. Lategahn, “Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme,” in _2010 IEEE Intelligent Vehicles Symposium_ , 2010, pp. 486–492.
  * [3] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” in _Readings in Computer Vision_ , M. A. Fischler and O. Firschein, Eds. San Francisco (CA): Morgan Kaufmann, 1987, pp. 726–740. [Online]. Available: https://www.sciencedirect.com/science/article/pii/B9780080515816500702
  * [4] G. Klein and D. Murray, “Parallel tracking and mapping for small ar workspaces,” in _2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality_ , 2007, pp. 225–234.
  * [5] E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,” vol. 3951, 07 2006.
  * [6] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardós, “Orb-slam: A versatile and accurate monocular slam system,” _IEEE Transactions on Robotics_ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [7] J. Engel, V. Koltun, and D. Cremers, “Direct sparse odometry,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 3, pp. 611–625, 2018.
  * [8] J. Engel, T. Schöps, and D. Cremers, “LSD-SLAM: Large-scale direct monocular SLAM,” in _European Conference on Computer Vision (ECCV)_ , September 2014.
  * [9] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “On-manifold preintegration for real-time visual–inertial odometry,” _IEEE Transactions on Robotics_ , vol. 33, no. 1, pp. 1–21, 2016.
  * [10] T. Qin, P. Li, and S. Shen, “Vins-mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Transactions on Robotics_ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [11] S. Wang, R. Clark, H. Wen, and N. Trigoni, “Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 2043–2050.
  * [12] A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in _2015 IEEE International Conference on Computer Vision (ICCV)_ , 2015, pp. 2758–2766.
  * [13] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, pp. 1735–80, 12 1997.
  * [14] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni, “Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [15] C. Chen, S. Rosa, Y. Miao, C. X. Lu, W. Wu, A. Markham, and N. Trigoni, “Selective sensor fusion for neural visual-inertial odometry,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 542–10 551.
  * [16] C. Chen, S. Rosa, C. X. Lu, B. Wang, N. Trigoni, and A. Markham, “Learning selective sensor fusion for state estimation,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [17] F. Xue, X. Wang, S. Li, Q. Wang, J. Wang, and H. Zha, “Beyond tracking: Selecting memory and refining poses for deep visual odometry,” in _IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 8575–8583.
  * [18] X. Kuo, C. Liu, K. Lin, E. Luo, Y. Chen, and C. Lee, “Dynamic attention-based visual odometry,” in _The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 5753–5760.
  * [19] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, “Unsupervised learning of depth and ego-motion from video,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 6612–6619.
  * [20] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [21] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3828–3838.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [23] R. Peng, R. Wang, Y. Lai, L. Tang, and Y. Cai, “Excavating the potential capacity of self-supervised monocular depth estimation,” in _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021, pp. 15 540–15 549.
  * [24] J. Bian, Z. Li, N. Wang, H. Zhan, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth and ego-motion learning from monocular video,” _Advances in neural information processing systems_ , vol. 32, 2019\.
  * [25] L. Wang, Y. Wang, L. Wang, Y.-W. Zhan, Y. Wang, and H. Lu, “Can scale-consistent monocular depth be learned in a self-supervised scale-invariant manner?” _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 12 707–12 716, 2021.
  * [26] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3d packing for self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2485–2494.
  * [27] Y. Almalioglu, M. Turan, M. R. U. Saputra, P. P. de Gusmão, A. Markham, and N. Trigoni, “Selfvio: Self-supervised deep monocular visual–inertial odometry and depth estimation,” _Neural Networks_ , vol. 150, pp. 119–136, 2022.
  * [28] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE signal processing magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [29] P. Wei, G. Hua, W. Huang, F. Meng, and H. Liu, “Unsupervised monocular visual-inertial odometry network,” in _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2021, pp. 2347–2354.
  * [30] A. Wong, X. Fei, S. Tsuei, and S. Soatto, “Unsupervised depth completion from visual inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1899–1906, 2020.
  * [31] F. Ma, G. V. Cavalheiro, and S. Karaman, “Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 3288–3295.
  * [32] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [33] T. Shen, Z. Luo, L. Zhou, H. Deng, R. Zhang, T. Fang, and L. Quan, “Beyond photometric loss for self-supervised ego-motion estimation,” _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 6359–6365, 2019\.