# Deep IMU Bias Inference for Robust
Visual-Inertial Odometry with Factor Graphs

Russell Buchanan1, Varun Agrawal2, Marco Camurri1, Frank Dellaert2, Maurice
Fallon1 1 Oxford Robotics Institute, University of Oxford, UK {russell,
mcamurri<EMAIL_ADDRESS>Institute for Robotics and Intelligent
Machines, College of Computing, Georgia Institute of Technology {varunagrawal,
<EMAIL_ADDRESS>divergeu

###### Abstract

Visual Inertial Odometry (VIO) is one of the most established state estimation
methods for mobile platforms. However, when visual tracking fails, VIO
algorithms quickly diverge due to rapid error accumulation during inertial
data integration. This error is typically modeled as a combination of additive
Gaussian noise and a slowly changing bias which evolves as a random walk. In
this work, we propose to train a neural network to learn the true bias
evolution. We implement and compare two common sequential deep learning
architectures: LSTMs and Transformers. Our approach follows from recent
learning-based inertial estimators, but, instead of learning a motion model,
we target IMU bias explicitly, which allows us to generalize to locomotion
patterns unseen in training. We show that our proposed method improves state
estimation in visually challenging situations across a wide range of motions
by quadrupedal robots, walking humans, and drones. Our experiments show an
average 15% reduction in drift rate, with much larger reductions when there is
total vision failure. Importantly, we also demonstrate that models trained
with one locomotion pattern (human walking) can be applied to another
(quadruped robot trotting) without retraining.

##  I Introduction

State estimation for lightweight, mobile systems is a fundamental problem in
robotics. Visual Inertial Odometry (VIO) is a common solution due to the small
size and low cost of cameras and IMUs. The main weak-point of VIO is that when
visual feature tracking fails, only the Inertial Measurement Unit (IMU) can be
used.

Inexpensive and miniaturized micro-electromechanical systems (MEMS) IMUs have
become ubiquitous in robotics and pedestrian tracking [1, 2, 3, 4], however
IMU-only state estimation is severely affected by drift. This drift is the
result of the accumulation of various errors in IMU integration collectively
modeled as a combination of additive zero-mean Gaussian noise and a slowly
changing bias. As a result, estimation which relies purely on IMU measurement
integration is not feasible for more than a few seconds due to explosive
accumulation of drift.

VIO is effective so long as visual features are available because they
constrain the drift of IMU integration [5] and estimate the biases. When
visual tracking fails completely, the system relies purely on IMU integration.
When there is error in visual tracking, the estimator may update the bias
estimates to make sense of the data. In other words, the estimated bias is not
necessarily related to the underlying physical process, but a quantity that
minimizes the residuals. If there were another way to infer the IMU biases,
these problems could be diminished.

Figure 1: Two of the platforms used in this work. Left: handheld device
including a SevenSense AlphaSense visual-inertial sensor. Right: the same
device mounted on a Boston Dynamics Spot quadruped. An IMU bias model from a
neural network trained on handheld data was used to infer the biases on the
Spot dataset without any retraining — despite the significantly different
motion pattern. The lidar was not used in these experiments.

Most VIO works have focused on improving visual tracking, however in this work
we take a novel approach of improving IMU bias modeling. We propose a new
method which uses deep learning to estimate IMU biases directly from IMU
measurements and past biases. We train a neural network to learn the evolution
process of the biases of a specific IMU rather than assuming Brownian Motion.
As a result, our method is device specific, not locomotion specific, unlike
similar IMU learning approaches, and does not require periodic motion. Our
contributions can be listed as follows:

  * •

A neural network capable of estimating IMU biases from a history of
measurements and biases. To the best of the authors’ knowledge, this is the
first method capable of explicitly inferring the bias evolution of an IMU.

  * •

A performance comparison of two different network implementations (LSTM and
Transformer) and their integration as unary factors into a state estimator
based on factor graphs, for improved estimation in visually challenging
scenarios.

  * •

Real-world experiments on three different platforms with different types of
motions: pedestrian handheld, quadrupedal robot and drones. To the best of the
authors’ knowledge, this is the first work that demonstrates an IMU learned
model trained on one locomotion modality and tested on another (handheld to
quadrupedal).

An additional minor contribution is the development of a ROS compatible, open-
source tool for calibrating an IMU using the Allan Variance
method111https://github.com/ori-drs/allan_variance_ros.

##  II Related Work

In this section, we summarize the growing field of inertial learning from
which our work follows. In Section II-A, we discuss methods which learn motion
models from inertial data, while Section II-B covers methods which learn IMU
noise models primarily for drone state estimation.

###  II-A Learning Inertial Motion Models

Recent works have trained neural networks with IMU data to learn motion models
(typically of pedestrians) and to output estimates of velocity directly from
IMU measurements.

The first data driven Inertial Navigation System (INS) was IoNet by Chen et
al. [6] which inferred 2D displacement and orientation change from buffered
IMU data. Later, Yan et al. [7] proposed RoNIN, a similar network which
inferred 2D velocity and orientation from raw IMU data. Liu et al. [8]
proposed TLIO, a method to infer 3D motion and estimate high-fidelity
trajectories in a filtering framework. They used an Extended Kalman Filter
(EKF) to estimate the full 6 DoF state. Process updates were performed by
traditional IMU mechanization while measurement updates from the network were
used as relative position measurements. This allowed the filter to implicitly
estimate IMU biases.

In our prior work, we adapted the approach of [8] to incorporate additional
sensors such as cameras, lidar or legged robot kinematics in a factor graph
[9]. However, this method, like the ones discussed above, was based on
learning a motion model and was therefore susceptible to failure if applied
outside of the training domain. For example, in Fig. 2 we show results of TLIO
which was trained with handheld walking data on flat ground. As shown in Fig.
2 a), the velocity estimates are reasonable for IMU-only odometry. In Fig. 2
b), we applied the same network to stair climbing, which was not present in
the training set and as a result, the $z$ velocity estimation fails
completely. In Fig. 2 c) plot we used the network with a quadruped and the
model fails completely. This motivates the need for methods which are more IMU
specific rather than locomotion specific.

###  II-B Learning IMU Noise Models

A different approach to velocity estimation is learning IMU noise models. In
this case, the IMU measurements are passed through a network trained to
“denoise” them and output an estimate of the perfect IMU measurements. These
denoised signals can then be used directly as input to a VIO pipeline.

Figure 2: Top: TLIO [8] trained on handheld data on flat ground, tested on
flat ground. Middle: same network tested on a sequence in which the person
walks up and down stairs. Because stair climbing was not present in the
training set, inference fails significantly (Note $z$ velocity estimates).
Bottom: when applied to a quadrupedal robot, because of the completely
different locomotion modality, TLIO is unable to estimate any position.

Brossard et al. [10] used a Convolutional Neural Network (CNN) to denoise a
gyroscope using the output to correct for the true angular velocity. The CNN
used dilated convolutions to increase the temporal coverage over longer
sequences. Zhang et al. [11] trained a Recurrent Neural Network (RNN) to
denoise both gyroscope and accelerometer measurements. Their network used IMU
measurements as input to estimate a corrected measurement which, when
integrated, reduced pose error. This eliminated the need for the model to
learn the underlying mechanization equations. Similarly, Steinbrener et al.
[12] performed denoising on IMU measurements, comparing LSTM and Transformer
architectures, finding the LSTM to be more effective.

The main drawback of these methods is that they do not distinguish between
different noise sources and as a result, it is not clear what the network has
learned to remove from the IMU measurements. For example, most results were
primarily demonstrated on drones which, when flying, introduce high frequency
vibrations affecting the IMU measurements. It is unclear if the noise
characteristics being estimated was caused by the vibrations, the internal
state of the IMU (i.e. the true bias), the motion of the drone, or a
combination of the above. By explicitly modeling the IMU bias, in this paper
we seek a motion independent method with more explainable outputs.

##  III Problem Statement

We aim to estimate the trajectory of a mobile platform equipped with an IMU
and a stereo camera using sliding window based smoothing. The relevant
reference frames of the platform are shown in Fig. 3 and includes an Earth-
fixed world frame $\mathtt{W}$, the platform-fixed base frame $\mathtt{{B}}$,
left and right camera frames $\mathtt{C}_{0},\mathtt{C}_{1}$, and IMU sensor
frame $\mathtt{I}$.

Figure 3: Reference frames convention for the handheld device. The world frame
$\mathtt{W}$ is a fixed frame, while the base $\mathtt{{B}}$, camera optical
$\mathtt{C}$, IMU $\mathtt{I}$, and lidar $\mathtt{L}$ frames are attached to
the moving device.

###  III-A Definition of State Vector and Measurements

The state of the platform at time $t_{i}$ is defined as:

$\boldsymbol{x}_{i}\triangleq\left[\mathbf{R}_{i},\mathbf{p}_{i},\mathbf{v}_{i},\mathbf{b}^{g}_{i}\,\;\mathbf{b}^{a}_{i}\right]\in\mathrm{SO}(3)\times\mathbb{R}^{12}$
(1)

where: $\mathbf{R}_{i}=\mathbf{R}_{\mathtt{{W}}\mathtt{{B}}}(t_{i})$ is the
orientation of $\mathtt{{B}}$ with respect to $\mathtt{W}$,
$\mathbf{p}_{i}=\tensor[_{\mathtt{{W}}}]{\mathbf{p}}{{}_{\mathtt{{W}}\mathtt{{B}}}}(t_{i})$
is the position,
$\mathbf{v}_{i}=\tensor[_{\mathtt{{B}}}]{\mathbf{v}}{{}_{\mathtt{{W}}\mathtt{{B}}}}(t_{i})$
is the linear velocity, and
$\mathbf{b}^{g}_{i}=\tensor[_{\mathtt{{I}}}]{\mathbf{b}}{{}^{g}}(t_{i}),\;\mathbf{b}^{a}_{i}=\tensor[_{\mathtt{{I}}}]{\mathbf{b}}{{}^{a}}(t_{i})$
are the IMU gyroscope and accelerometer biases, respectively.

We indicate the set of all states in the window as
$\mathcal{X}_{k}=\\{\boldsymbol{x}_{i}\\}_{i\in\mathsf{K}_{k}}$ where
$\mathsf{K}_{k}$ are all the keyframe indices up to $t_{k}$. Similarly, the
measurements within the window are $\mathcal{Z}_{k}=\\{{\cal I}_{ij},{\cal
C}_{i}\\}_{i,j}\in\mathsf{K}_{k}$, where ${\cal I}_{ij}$ are the IMU
measurements between two camera keyframes (with $i=j-1$), while ${\cal C}_{i}$
include the stereo image pairs at time $t_{i}$.

###  III-B Maximum-A-Posteriori (MAP) Estimation

We maximize the likelihood of the measurements ${\cal Z}_{k}$, given the
history of states ${\cal X}_{k}$,

$\mathcal{X}^{*}_{k}=\operatorname*{arg\,max}_{\mathcal{X}_{k}}p(\mathcal{X}_{k}|\mathcal{Z}_{k})\propto
p(\mathcal{X}_{0})p(\mathcal{Z}_{k}|\mathcal{X}_{k})$ (2)

The measurements are formulated as conditionally independent and corrupted by
white Gaussian noise. Therefore, (2) can be expressed as the following least
squares minimization:

$\mathcal{X}^{*}_{k}=\operatorname*{arg\,min}_{\mathcal{X}_{k}}\|\mathbf{r}_{0}\|^{2}_{\Sigma_{0}}+\sum_{i\in\mathsf{K}_{k}}\Big{(}\|\mathbf{r}_{{\cal
I}_{ij}}\|^{2}_{\Sigma_{{\cal I}_{ij}}}+\\\
\sum_{\ell\in\mathsf{M}_{i}}\|\mathbf{r}_{\boldsymbol{x}_{i},\boldsymbol{m}_{\ell}}\|^{2}_{\Sigma_{\boldsymbol{x}_{i},\boldsymbol{m}_{\ell}}}+\|\mathbf{r}_{{\mathbf{b}}^{a}_{j}}\|^{2}_{\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{a}}}+\|\mathbf{r}_{{\mathbf{b}}^{g}_{j}}\|^{2}_{\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{g}}}\Big{)}$
(3)

where each term is the residual associated to a measurement type, weighted by
the inverse of its covariance matrix, and will be detailed in Section IV.

Figure 4: Proposed factor graph framework with learned IMU estimates. A neural
network $\mathcal{F}_{\theta}$ trained on ground truth IMU biases takes a
window of past IMU measurements ${\cal I}_{mj}$ and previous bias estimates
$\mathbf{b}_{ni}$ from the optimizer and outputs the current bias estimate
$\hat{\mathbf{b}}_{j}$. This is integrated into the factor graph as unary
factor (magenta). This helps where visual features cannot be tracked reliably
(right side of figure where landmarks are absent).

##  IV Factor Graph Formulation

A factor graph can be used to graphically represent (3) as in Fig. 4. With
slight abuse of notation, the IMU biases are shown separately from the state
nodes $\boldsymbol{x}_{i}$ to highlight our contribution. In addition to the
prior factors (black), the graph includes: IMU preintegration (blue), visual
tracking (yellow) and our novel deep IMU bias (purple) factors. For
convenience, the first two are briefly reported in this section, while the
last one is detailed in Section V.

###  IV-A Preintegrated IMU Factors

We follow the standard manner of IMU measurement integration from [5] to
constrain the pose, velocity, and biases between two consecutive nodes of the
graph. The residual has the following form:

$\mathbf{r}_{{\cal
I}_{ij}}=\left[\mathbf{r}^{\mathsf{T}}_{\Delta\mathbf{R}_{ij}},\mathbf{r}^{\mathsf{T}}_{\Delta\mathbf{v}_{ij}},\mathbf{r}^{\mathsf{T}}_{\Delta\mathbf{p}_{ij}},\mathbf{r}_{\mathbf{b}^{a}_{ij}},\mathbf{r}_{\mathbf{b}^{g}_{ij}}\right]$
(4)

For a detailed definition of the above residuals, see [5].

###  IV-B Stereo and Mono Landmark Factors

A visual landmark in Euclidean space $\boldsymbol{m}_{\ell}\in\mathbb{R}^{3}$
is projected onto the image plane by the function
$\pi:\mathrm{SE}(3)\times\mathbb{R}^{3}\mapsto\mathbb{R}^{2}$, given the
platform pose
$\mathbf{T}_{i}=\\{\mathbf{p}_{i},\mathbf{R}_{i}\\}\in\mathrm{SE}(3)$. Given a
landmark $\boldsymbol{m}_{\ell}$ and its coordinates
$(u_{\ell},v_{\ell})\in\mathbb{R}^{2}$ on the image plane, the residual at
state is computed as:

$\mathbf{r}_{\boldsymbol{x}_{i},\boldsymbol{m}_{\ell}}=\left(\begin{array}[]{c}\pi_{u}^{L}(\mathbf{T}_{i},\boldsymbol{m}_{\ell})-u^{L}_{i,\ell}\\\
\pi_{u}^{R}(\mathbf{T}_{i},\boldsymbol{m}_{\ell})-u^{R}_{i,\ell}\\\
\pi_{v}(\mathbf{T}_{i},\boldsymbol{m}_{\ell})-v_{i,\ell}\end{array}\right)$
(5)

where $(u^{L},v),(u^{R},v)$ are the pixel locations of the landmark.
$\Sigma_{\boldsymbol{m}}$ is computed using an uncertainty of
$0.25\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}$
and the Dynamic Covariance Scaling robust loss function as in [13].

##  V Deep IMU Bias Factors

A MEMS IMU measures its proper acceleration (e.g., equal to Earth’s gravity at
rest) $\tilde{\mathbf{a}}\in\mathbb{R}^{3}$ and rotational rate
$\tilde{\boldsymbol{\omega}}\in\mathbb{R}^{3}$. The absolute linear
acceleration $\mathbf{a}$ (e.g., null at rest) and the rotation rate
$\boldsymbol{\omega}$ of a body expressed in an Earth-fixed coordinate frame
$\mathtt{W}$ can be recovered as follows:

$\displaystyle\begin{split}\tilde{\mathbf{a}}(t)=&\mathbf{R}_{\mathtt{{W}}\mathtt{{I}}}^{\mathsf{T}}(t)(\mathbf{a}(t)-\tensor[_{\mathtt{W}}]{\mathbf{g}}{})+{\mathbf{b}}^{a}(t)+\boldsymbol{\eta}^{a}(t)\\\
\tilde{\boldsymbol{\omega}}(t)=&\boldsymbol{\omega}(t)+{\mathbf{b}}^{g}(t)+\boldsymbol{\eta}^{g}(t)\end{split}$
(6)

Where $\mathbf{R}_{\mathtt{{W}}\mathtt{{I}}}\in\mathrm{SO}(3)$ is the absolute
orientation of the IMU and $\tensor[_{\mathtt{W}}]{\mathbf{g}}{}$ is the
acceleration due to gravity expressed in $\mathtt{W}$. The quantities
$\boldsymbol{\eta}^{a}$ and $\boldsymbol{\eta}^{g}$ represent additive noise
present in all IMUs and is modeled as zero-mean Gaussian distributions.

###  V-A Bias Noise Model

The IMU biases ${\mathbf{b}}^{a}$ and ${\mathbf{b}}^{g}$ are due to physical
properties of the IMU. They change with each power cycle and continue to
change slowly during operation [14]. Typically, their evolution is modeled as
a Brownian noise process, whose derivatives are sampled from zero-mean
Gaussian distributions:

$\dot{\mathbf{b}}^{a}(t)=\boldsymbol{\eta}^{ba}\qquad\dot{\mathbf{b}}^{g}(t)=\boldsymbol{\eta}^{bg}$
(7)

This can be re-written in discrete time by integrating between two time steps
$[t_{i},t_{j}]$:

${\mathbf{b}}^{a}_{j}={\mathbf{b}}^{a}_{i}+\boldsymbol{\eta}^{bad}\qquad{\mathbf{b}}^{g}_{j}={\mathbf{b}}^{g}_{i}+\boldsymbol{\eta}^{bgd}$
(8)

Where $\boldsymbol{\eta}^{bad}$ and $\boldsymbol{\eta}^{bgd}$ are discrete
zero mean Gaussian distributions with covariance $\mathbf{\Sigma}^{bad}$ and
$\mathbf{\Sigma}^{bgd}$ respectively [5].

###  V-B Learning the Bias Process Model

The Brownian noise model for IMU biases in (8) is an approximation which does
not hold for long periods of time. In practice, the true underlying dynamics
are a highly nonlinear function depending on vibrations, impacts, and physical
properties of the device [14]. Thus, modeling the IMU dynamics is a problem
that lends itself to deep learning as it allows us to approximate highly
nonlinear functions. To this end, instead of (8), we propose a deep neural
network ${\cal F}$ with parameters $\theta$ defined as follows:

${\cal F}_{\theta}:\left({\cal
I}_{mj},{\mathbf{b}}^{a}_{ni},{\mathbf{b}}^{g}_{ni}\right)\mapsto(\hat{\mathbf{b}}^{a}_{j},\hat{\mathbf{b}}^{g}_{j})$
(9)

where the inputs are a buffer of IMU measurements ${\cal I}_{mj}$ between
times $t_{m}$ and $t_{j}$, and the previous bias values
${\mathbf{b}}^{a}_{ni},{\mathbf{b}}^{g}_{ni}$ between $t_{n}$ and $t_{i}$. The
network outputs $(\hat{\mathbf{b}}^{a}_{j},\hat{\mathbf{b}}^{g}_{j})$ are the
estimates of the IMU bias value at time $t_{j}$. Without loss of generality,
in this section we assume the output of the network is generated at the camera
keyframe rate.

###  V-C Deep IMU Bias Factors

We incorporate the bias estimates from the network into a factor graph-based
state estimator. The estimates are modeled as unary factors on the bias state,
as shown in Fig. 4. The last two residuals of (3),
$\mathbf{r}_{{\mathbf{b}}^{a}_{j}}$ and $\mathbf{r}_{{\mathbf{b}}^{g}_{j}}$
correspond to:

$\mathbf{r}_{{\mathbf{b}}^{a}_{j}}={\mathbf{b}}^{a}_{j}-\hat{\mathbf{b}}^{a}_{j}\quad\quad\mathbf{r}_{{\mathbf{b}}^{g}_{j}}={\mathbf{b}}^{g}_{j}-\hat{\mathbf{b}}^{g}_{j}$
(10)

The covariances $\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{a}}$ and
$\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{g}}$ were tuned to fixed values for our
experiments, with
$\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{a}}=\mathbf{I}_{3}\cdot
2.5\mathrm{e}^{-3}$ and
$\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{g}}=\mathbf{I}_{3}\cdot
2.5\mathrm{e}^{-5}$. In future work, we intend to train the networks to
provide measurement uncertainty estimates as in [8, 9].

Figure 5: LSTM architecture. A IMU data window $\mathcal{I}_{mj}$ of size $w$
(with $m=j-w$) and the previous bias $\mathbf{b}_{i}$ are first normalized
then passed to the LSTM. The hidden state is preserved for the next inference
step and the output is passed through a fully connected layer to predict a
bias. Figure 6: Transformer architecture. A history of $l$ IMU windows
$\mathcal{I}_{mj}$ and biases $\mathbf{b}_{i}$ are concatenated and summed
with a positional encoding before being passed to the Transformer.

##  VI Network Architecture

Since our learning task involves sequential data, two common network
architectures are well suited when implementing (9). The first one is based on
Long Short-Term Memory (LSTM) [15] which is lightweight (see Fig. 5) and the
second is based on Transformers [16] which have been shown to have better
performance for longer-term sequences (see Fig. 6).

###  VI-A LSTM

The input to the LSTM is a single window of size $w$ of IMU measurements
${\cal I}_{mj}$ (with $m=j-w$) and the previous bias estimate $\mathbf{b}_{i}$
(i.e., $t_{n}=t_{i}$), which comes from the factor graph optimizer. This
exposes the network to bias estimates resulting from the fusion of additional
sensors. These are normalized then passed to the LSTM with states
$\mathbf{h}_{i}$ and $\mathbf{c}_{i}$ which are preserved for the next bias
estimate. In this way, the LSTM $w$ batches of IMU measurements while the
memory can observe the bias evolution over time. One limitation to this method
is that, over a long trajectory, the LSTM will eventually forget information
from old inputs.

###  VI-B Transformer

The Transformer input is a history of $l$ windows of IMU measurements and
biases (i.e., $m=j-l\cdot w,n=l$). Similar to the LSTM, biases added to the
history come from the estimator’s optimizer. A history of information allows
the Transformer attention mechanism to recall older information.

###  VI-C Loss Function

We use the Mean Square Error (MSE) as a loss function:

${\cal
L}(\mathbf{b},\hat{\mathbf{b}})=\frac{1}{n}\sum^{n}_{k=1}\|\mathbf{b}_{k}-\hat{\mathbf{b}}_{k}\|^{2}$
(11)

where two separate instances of the network are trained with (11) for
accelerometer and gyroscope biases, respectively.

##  VII Network Implementation

###  VII-A Datasets

The datasets used for training and experiments are listed in Table I. The
first is the Newer College Multi-Camera Dataset (NCD-Multi) [17] which uses an
AlphaSense inertial and multi-camera sensor (see Fig. 1), although we only use
the front facing stereo cameras. NCD-Multi was collected by a human operator
walking while carrying the device inside New College, Oxford. An additional
non-public sequence was collected using the same device in a limestone mine
[18].

The second dataset, Spot, was recorded to demonstrate generalization to
motions patterns unseen in training. We placed the same sensor payload used in
NCD-Multi on a quadruped robot (see Fig. 1 Right) and recorded two sequences
only for testing.

Finally, we present results with the public EuRoC dataset [19]. We included
this dataset to show results alongside similar methods and to demonstrate
application to a very different platform - a drone.

TABLE I: Datasets Used for Experiments.

Dataset | Duration [s] | Length [m] | IMU | Locom.
---|---|---|---|---
NCD-Multi | 3163 | 4512 | BMI085* | Handheld
Spot | 515 | 399 | BMI085* | Quadruped
EuRoC | 1349 | 893 | ADIS16448 | Drone

*Same IMU device.

###  VII-B Training

We trained one LSTM and one Transformer each for the public datasets (NCD-
Multi and EuRoC), for a total of four models. For each dataset, we selected
hard sequences for testing (see VIII-A) and the remaining were split 75:25 for
training and validation. The Spot dataset was only used for testing because,
as detailed in Section VIII, we demonstrate generalization to different
locomotion modalities by training a model on NCD-Multi and testing it on the
Spot dataset.

For training, we used teacher forcing [20], providing the network with ground
truth biases. The ground truth biases for NCD-Multi were estimated using the
lidar and a high resolution 3D model of the environment while the EuRoC
dataset provides IMU bias estimates. We added noise to the ground truth biases
which was sampled from a zero-mean Gaussian distribution with covariances
$\mathbf{\Sigma}^{bad}$ and $\mathbf{\Sigma}^{bgd}$ estimated from Allan
Variance analysis.

As in [8], we rotate the IMU measurement windows ${\cal I}_{mj}$ into a
gravity aligned frame. This prevents the effect of gravity from significantly
changing the apparent accelerometer biases.

We used the Adam optimizer with a learning rate of $10^{-5}$ and batch size
$32$. Training lasted 200 epochs for both LSTM and Transformer with the model
minimizing validation error used. Both networks take $\sim
2\text{\,}\mathrm{h}\text{/}$ to train on a desktop computer with one Nvidia
Titan X with $12\text{\,}\mathrm{GB}\text{/}$ of memory.

###  VII-C Networks Models

####  VII-C1 LSTM

We used a 2-layer single direction LSTM with hidden state size of $256$. We
found that a larger network provided no improvement in performance while a
lighter-weight LSTM is capable of running online with several inferences per
second on a standard laptop with low-grade GPU. After testing different
options, we found that $1\text{\,}\mathrm{s}\text{/}$ of IMU data (i.e.,
$w=10$) and an inference rate of $2\text{\,}\mathrm{Hz}\text{/}$ (50% data
overlap between consecutive inferences) to give the best performance and used
these settings for all experiments.

####  VII-C2 Transformer

We used an 8 headed Transformer with 2 encoder and decoder layers and an
embedding size of 512. Similar to LSTM, we used $1\text{\,}\mathrm{s}\text{/}$
IMU windows with a maximum history size of $l=100$. This was based on analysis
of the Allan Variance plots which found the IMU bias instability to dominate
noise generally around $100\text{\,}\mathrm{s}\text{/}$ sampling times.
Inference was kept at $1\text{\,}\mathrm{Hz}\text{/}$ since using a history of
IMU windows makes overlapping of input data unnecessary.

Figure 7: Visual challenges in the NCD-Multi Dataset. Top Row: Images from
NCD-Multi Stairs sequence with door opening obscuring the cameras shown on the
left and tight spaces of about 1.$5\text{\,}\mathrm{m}\text{/}$ wide and a
mirror shown on the right. Bottom Row: Images from NCD-Multi Mine sequence
with darkness and the challenge of camera being held up close to a wall.

##  VIII Experimental Results

In this section, we describe the experiments and results of the proposed state
estimator. We report relative position and orientation errors as described in
[21]. We compare our method which uses bias predictions as in Fig. 4 to our
baseline method, which consists of the same factor graph but without the unary
factor described in Section IV.

###  VIII-A Handheld Experiments

There are many common situations or environments in which visual odometry
systems degrade or even fully fail such as door-opening, narrow spaces or fast
rotations. We selected several sequences from NCD-Multi which exhibit these
challenges to test our method. These sequences were only used for testing and
contained situations unseen in training.

TABLE II: $10\text{\,}\mathrm{m}\text{/}$ Relative Translation/Rotation Error
[m]/[$\mathrm{\SIUnitSymbolDegree}\text{/}$]

Experiment | VIO baseline | LSTM | Transformer
---|---|---|---
NCD-Multi Stairs* | 0.23 / 3.22 | 0.19 / 2.88 |  0.17 / 2.70
NCD-Multi Mine | 0.46 / 2.59 |  0.33 / 1.71 | 0.42 / 2.26
NCD-Multi Quad | 0.35 / 1.38 | 0.28 / 1.36 |  0.25 / 1.29
Spot Easy | 0.31 / 1.21 |  0.27 / 1.02 | 0.31 / 0.88
Spot Hard | 0.38 / 1.57 | 0.34 /1.42 |  0.30 / 1.42

*$5\text{\,}\mathrm{m}\text{/}$ RPE due to shorter total distance traveled.

####  VIII-A1 NCD-Multi Stairs

This sequence included narrow spaces, mirrors, door opening and large
rotations as shown in Fig. 7. As a result, the baseline stereo visual odometry
struggled and lost track of all features several times.

We show in Fig. 8 that the learned bias predictions reduced relative pose
errors. This is most clearly seen in the magnified area where the door opening
caused a significant drop in tracked visual features. Our proposed approach
improves upon pure IMU-only mechanization in these situations where visual
feature tracking fails.

####  VIII-A2 NCD-Multi Mine

The Mine sequence was recorded in an abandoned limestone mine in Corsham, UK.
It was dark and dusty and the handheld platform was intentionally held up very
close to walls as shown in Fig. 7 on several occasions. This was also the
longest trajectory.

The results can be seen in Fig. 9, where we highlight two particular sections
in which the baseline VIO became unstable. In one section the camera faced a
wall for several seconds and another involved large rotations in the dark. We
see that by adding the bias prediction, trajectories are smoother and have
lower error overall.

Figure 8: NCD-Multi Stairs results. Top:  Position of the baseline VIO, LSTM
and Transformer methods on $xy$-plane. We magnify the door opening section of
the trajectory where the baseline clearly diverges due to loss of visual
feature tracking and poor bias estimation. Middle:  Position of the trajectory
for the $z$-axis. Bottom:  $5\text{\,}\mathrm{m}\text{/}$ RPE for each
algorithm.

####  VIII-A3 NCD-Multi Quad

This sequence is long and in a large-scale open space and includes several
dynamic motions when the handheld platform was shaken. Our method reduces
error compared to the baseline. A summary of these results and for the other
NCD-Multi sequences is provided in Table II.

###  VIII-B Quadruped Robot Experiments

For the Spot dataset, the same handheld sensor used in the NCD-Multi dataset
was mounted on a Boston Dynamics Spot (See Fig. 1) which was then teleoperated
around a courtyard in two separate trajectories. The Easy trajectory is
entirely on flat ground in good lighting conditions while the Hard trajectory
includes transitions from well-lit to shady areas and walking on small ramps.

For these experiments, both the LSTM and Transformer were trained using only
handheld data. There was no fine-tuning for the quadrupedal robot. The
information used for training and testing was therefore exactly the same as in
Fig. 2. Numerical results are provided in Table II. This demonstrates that our
method is independent from the locomotion modality and that the model can
generalize across different modalities.

Figure 9: Results from the Mine sequence. Top:  Position on $xy$-plane of the
baseline VIO, LSTM and Transformer methods. We magnify a part of the sequence
where the camera was held facing and very close to a wall. The baseline
diverges due to loss of visual feature tracking and poor bias estimation. We
also highlight a section with large rotations where the baseline has multiple
divergences. Middle:  Position on $z$-axis. Bottom:
$10\text{\,}\mathrm{m}\text{/}$ RPE for each algorithm.

###  VIII-C Drone Experiments

Two similar works to ours are Zhang et al. [11] and Brosard et al [10]. Zhang
et al. use an LSTM to denoise accelerometer and gyroscope measurements which
are then passed to their visual-inertial pipeline based on [22]. Brossard et
al. use a CNN to denoise gyroscope measurements only which are then passed to
Open-VINS [3]. Since both works present their results using the EuRoC dataset
[19], we also use this dataset to demonstrate that our method can be applied
to drones. Because the baselines were evaluated differently, drawing
conclusions from direct comparisons is difficult. Therefore, we also show
relative improvement for each method.

Tables III and IV summarize these results in terms of absolute error. Note
that [11] reported positional RMSE while [10] reported orientation RMSE and
therefore we report both. For the V101 sequence, we use the re-computed ground
truth from [3] as did Brossard et al. [10].

Brossard et al. additionally reported relative orientation errors by averaging
$7\text{\,}\mathrm{m}\text{/}$, $21\text{\,}\mathrm{m}\text{/}$ and
$32\text{\,}\mathrm{m}\text{/}$ relative orientation error for all
trajectories. Averaged over the three distances they report
$1.59\text{\,}\mathrm{\SIUnitSymbolDegree}\text{/}$ while we achieve
$1.41\text{\,}\mathrm{\SIUnitSymbolDegree}\text{/}$ and
$1.36\text{\,}\mathrm{\SIUnitSymbolDegree}\text{/}$ for LSTM and Transformer
respectively.

###  VIII-D Illustrative Experiments

We present two additional experiments to further analyze our system’s
performance. In both cases we used the Spot Easy trajectory and models trained
with only handheld data.

####  VIII-D1 Blackout

We “black out” the images for $5\text{\,}\mathrm{s}\text{/}$, setting all
pixels to $0$. As shown in Fig. 10 there is an error spike during the blackout
where the estimator must rely on IMU data only. We also experiment with
“locking” the IMU biases during the blackout rather than allowing the random
walk.

####  VIII-D2 Image Distortion

When there are errors in visual tracking, such as poor camera calibration or
wrong correspondences, the associated error can be erroneously attributed to
the IMU bias. Learned bias estimates can mitigate this by providing estimates
of the correct bias. To demonstrate this, we intentionally inject visual
tracking error by left shifting both stereo images by $10$ pixels for
$10\text{\,}\mathrm{s}\text{/}$. This is interpreted as an apparent rotation
on the $z$-axis, attributed by the system to a change in yaw gyro bias. In
Fig. 11 we plot the bias estimates during this time. Even though we initialize
all methods with the ground truth bias and use a robust loss function, each
method’s estimate of the gyroscope $z$ bias incorrectly drifts during the
period of image distortion.

TABLE III: EuRoC: Absolute Position Error & Improvement [$\mathrm{m}\text{/}$
(%)]

| Zhang et al. [11] | Ours
---|---|---
Seq. | Baseline | Method | Baseline | LSTM | Transformer
MH02 | 0.19 | 0.15 (21%) | 0.37 | 0.23 (38%) | 0.13 (65%)
MH04 | 0.15 | 0.14 (7%) | 0.33 | 0.25 (24%) | 0.25 (24%)
V101 | 0.08 | 0.07 (13%) | 0.17 | 0.08 (53%) | 0.08 (53%)
V103 | 0.27 | 0.15 (44%) | 0.27 | 0.27 (0%) | 0.17 (37%)
V202 | 0.11 | 0.10 (9%) | 0.29 | 0.23 (21%) | 0.10 (66%)

TABLE IV: EuRoC: Absolute Rotation Error & Improvement
[$\mathrm{d}\mathrm{e}\mathrm{g}$ (%)]

| Brossard et al. [10] | Ours
---|---|---
Seq. | Baseline | Method | Baseline | LSTM | Transformer
MH02 | 1.11 | 1.21 (-9%) | 3.21 | 3.10 (3%) | 2.86 (11%)
MH04 | 1.60 | 1.40 (13%) | 0.89 | 1.00 (-12%) | 0.76 (15%)
V101 | 0.80 | 0.80 (0%) | 2.56 | 1.17 (54%) | 1.66 (35%)
V103 | 2.32 | 2.25 (3%) | 4.78 | 3.54 (26%) | 1.87 (61%)
V202 | 1.85 | 1.81 (2%) | 3.78 | 2.53 (33%) | 1.31 (65%)

##  IX Discussion

In Section VIII-A we show our methods achieving reduced error compared to the
baseline in several visually challenging situations. On average the LSTM and
Transformer have reduced error in Table II by 15%, but in certain situations,
for example the camera held close to the wall in the Mine trajectory, the
improvement is over 300%. This is both because the learned estimates improve
bias estimation and because they stabilize the optimizer in situations where
error is incorrectly attributed to the IMU bias.

In Section VIII-B we take a model trained with handheld data and apply it to
quadrupedal data to demonstrate our method is locomotion agnostic.
Additionally, it should be noted the Stairs trajectory is the only handheld
trajectory with stair climbing and therefore that experiment also demonstrates
locomotion agnosticism. The reduced error of our method compared to the
baseline shows our method is sensor specific and not locomotion specific.

In the Section VIII-C drone experiments, our method has similar accuracy as
other methods. However, each method uses a different VIO baseline. Our method
improves upon our baseline VIO more so than other methods improve on their
baselines. Also, as the EuRoC dataset provides ground truth bias estimates, we
examine the accelerometer and gyroscope bias
$(\hat{\mathbf{b}}^{a},\hat{\mathbf{b}}^{g})$ errors and find that, on
average, error is reduced by (35%,23%) for LSTM and (32%,12%) for Transformer.

In the illustrative experiments in Section VIII-D we first present the
Blackout experiment where our proposed methods reduced error during IMU-only
estimation when no visual features could be tracked. Locking the bias did not
improve on the baseline. This is because the bias random walk is already
sufficiently constrained due to calibration. There is a second spike in error
for all methods after vision is restored. This is because the pose estimate is
suddenly corrected by a large amount, which locally appears as a relative
error spike.

In the Image Distortion experiment, a bias shift affects all methods but the
learned methods experience less error over the $10\text{\,}\mathrm{s}\text{/}$
and a faster recovery period. During distortion, the error of LSTM and
Transformer methods are 41% and 12% lower (RMSE) than the baseline.

LSTM and Transformer provide similar improvements to positional error with
Transformer slightly better at rotational error. This makes sense because the
gyroscope biases change more slowly and we expect the Transformer to be more
accurate with longer-term dependencies. In the distortion experiment the
Transformer instead performs more poorly than the LSTM which we believe is due
to the LSTM operating at $2\text{\,}\mathrm{Hz}\text{/}$ allowing it to react
to disturbances more quickly. In terms of model size the LSTM is superior with
learned models as small as $30\text{\,}\mathrm{MB}\text{/}$ compared to
$200\text{\,}\mathrm{MB}\text{/}$ for the transformer. Additionally, the LSTM
forward pass takes less than $6\text{\,}\mathrm{ms}\text{/}$ as compared to
$20\text{\,}\mathrm{ms}\text{/}$ for the Transformer.

Figure 10: Blackout Experiment: $5\text{\,}\mathrm{m}\text{/}$ RPE Performance
where the cameras were disabled for $5\text{\,}\mathrm{s}\text{/}$ (gray area)
during the Spot Easy trajectory. Mean $5\text{\,}\mathrm{m}\text{/}$ RPE was
$0.26\text{\,}\mathrm{m}\text{/}$, $0.28\text{\,}\mathrm{m}\text{/}$,
$0.22\text{\,}\mathrm{m}\text{/}$ and $0.20\text{\,}\mathrm{m}\text{/}$ for
the baseline, bias-lock, LSTM and Transformer methods respectively. Figure 11:
Image Distortion Experiment: At $100\text{\,}\mathrm{s}\text{/}$ of the Spot
Easy trajectory we apply a small distortion to both stereo images (pink area).
The baseline VIO develops a significant offset in $z$ bias as the error is
attributed to the gyroscope. LSTM and Transformer methods have reduced
magnitude of error and recovery time.

##  X Conclusion

We present a novel application of machine learning to inertial navigation
which is more interpretable than similar methods. By learning bias predictions
the proposed method is more generally applicable because it is not robot
specific and does not require periodic locomotion which we demonstrate with a
wide variety of experiments on quadrupeds, handheld sensors and drones. We
show that our method reduces $5\text{\,}\mathrm{m}\text{/}$ RPE of our
baseline VIO system by 15% on average in handheld and quadrupedal experiments
with RPE being reduced by as much as 300% in certain situations where vision
fails. In the future we will build on this work by integrating uncertainty
estimation into the model and experimenting with non-patterned locomotion such
as wheeled robots.

## ACKNOWLEDGMENT

This work was supported by the EPSRC ORCA Robotics Hub (EP/R026173/1), the EU
H2020 Project THING (Grant ID 780883) and a Royal Society University Research
Fellowship (Fallon). For the purpose of Open Access, the author has applied a
CC BY public copyright licence to any Author Accepted Manuscript (AAM) version
arising from this submission.

## References

  * [1] D. Wisth, M. Camurri, S. Das, and M. Fallon, “Unified multi-modal landmark tracking for tightly coupled lidar-visual-inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, p. 1004–1011, 2021.
  * [2] S. Zhao, H. Zhang, P. Wang, L. Nogueira, and S. Scherer, “Super Odometry: IMU-centric LiDAR-visual-inertial estimator for challenging environments,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 8729–8736.
  * [3] P. Geneva, K. Eckenhoff, W. Lee, Y. Yang, and G. Huang, “OpenVINS: A research platform for visual-inertial estimation,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 4666–4672.
  * [4] E. Foxlin, “Pedestrian tracking with shoe-mounted inertial sensors,” _IEEE Computer Graphics and Applications_ , vol. 25, no. 6, pp. 38–46, 2005\.
  * [5] C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “On-manifold preintegration for real-time visual-inertial odometry,” _IEEE Transactions on Robotics_ , vol. 33, no. 1, pp. 1–21, 2017.
  * [6] C. Chen, X. Lu, A. Markham, and N. Trigoni, “IoNet: Learning to cure the curse of drift in inertial odometry,” in _AAAI Conference on Artificial Intelligence_ , 2018, pp. 6468–6476.
  * [7] S. Herath, H. Yan, and Y. Furukawa, “RoNIN: Robust neural inertial navigation in the wild: Benchmark, evaluations, new methods,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 3146–3152.
  * [8] W. Liu, D. Caruso, E. Ilg, J. Dong, A. I. Mourikis, K. Daniilidis, _et al._ , “TLIO: Tight learned inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5653–5660, 2020.
  * [9] R. Buchanan, M. Camurri, F. Dellaert, and M. Fallon, “Learning inertial odometry for dynamic legged robot state estimation,” in _Conference on Robot Learning_ , vol. 164, 2022, pp. 1575–1584.
  * [10] M. Brossard, S. Bonnabel, and A. Barrau, “Denoising IMU gyroscopes with deep learning for open-loop attitude estimation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4796–4803, 2020.
  * [11] M. Zhang, M. Zhang, Y. Chen, and M. Li, “IMU data processing for inertial aided navigation: A recurrent neural network based approach,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 3992–3998.
  * [12] J. Steinbrener, C. Brommer, T. Jantos, A. Fornasier, and S. Weiss, “Improved state propagation through AI-based pre-processing and down-sampling of high-speed inertial data,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , 2022.
  * [13] D. Wisth, M. Camurri, and M. Fallon, “VILENS: Visual, inertial, lidar, and leg odometry for all-terrain legged robots,” _IEEE Transactions on Robotics_ , 2022, to appear.
  * [14] D. Titterton and J. Weston, _Strapdown Inertial Navigation Technology_. Institution of Engineering and Technology, 2004.
  * [15] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [16] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, _et al._ , “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [17] L. Zhang, M. Camurri, D. Wisth, and M. Fallon, “Multi-camera lidar inertial extension to the Newer College Dataset,” Tech. Rep., 2021. [Online]. Available: https://arxiv.org/abs/2112.08854
  * [18] L. Zhang, D. Wisth, M. Camurri, and M. Fallon, “Balancing the Budget: Feature selection and tracking for multi-camera visual-inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 1182–1189, 2022\.
  * [19] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, _et al._ , “The EuRoC micro aerial vehicle datasets,” _The International Journal of Robotics Research_ , 2016.
  * [20] R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” _Neural Computation_ , vol. 1, no. 2, pp. 270–280, 1989.
  * [21] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, “A benchmark for the evaluation of RGB-D SLAM systems,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2012, pp. 573–580.
  * [22] M. Zhang, X. Zuo, Y. Chen, Y. Liu, and M. Li, “Pose estimation for ground robots: On manifold representation, integration, reparameterization, and optimization,” _IEEE Transactions on Robotics_ , vol. 37, no. 4, pp. 1081–1099, 2021.