# IMU Bias Prediction with Neural Networks
for Robust Visual-Inertial Odometry

Russell Buchanan1, Varun Agrawal2, Marco Camurri1, Frank Dellaert2, Maurice
Fallon1 1 Oxford Robotics Institute, University of Oxford, UK {russell,
mcamurri<EMAIL_ADDRESS>Institute for Robotics and Intelligent
Machines, College of Computing, Georgia Institute of Technology {varunagrawal,
<EMAIL_ADDRESS>
###### Abstract

Visual Inertial Odometry is a powerful tool in robot navigation. However when
visual feature tracking fails, inertial integration becomes a critical weak
point of VIO. Previous work has focused on learning a motion model from the
IMU data which can be specific to the pattern of robot locomotion. In this
work, we instead propose to train a network to predict the immediate evolution
of the IMU biases. Our preliminary results show that learned bias predictions
can improve state estimation in visually challenging situations. Our
experiments demonstrate a 56% reduction in drift rate in these situations. We
also use this approach to enable in-motion initialization of VIO and show
faster bias convergence leading to less error.

##  I INTRODUCTION

Inertial Measurement Units (IMUs) have long been an important sensor for state
estimation, from naval navigation to extensive use in aerospace
industries[strapdown-ins]. Less expensive consumer-grade IMUs, particularly
those using Micro-electromechanical systems (MEMS) technology have become
ubiquitous in robotics and pedestrian tracking[Wisth_2021, super-odometry,
qin2017vins, Foxlin2005]. MEMS IMUs are small, lightweight and relatively
inexpensive. Additionally, unlike exteroceptive sensors such as cameras and
lidar, they are unaffected by external environmental effects such as darkness
or fog. However, IMU-only state estimation which relies purely on measurement
integration is not feasible for long durations due to explosive accumulation
of drift [strapdown-ins, Foxlin2005].

In recent years there have been several works investigating the use of machine
learning in IMU-only state estimation. Most methods train a deep neural
network to learn a model of the subject’s motion (e.g. a legged robot or a
human pedestrian) [Chen2018, Yan2019, liu2020tlio, buchanan-corl]. In these
works, the network uses its motion model to infer velocity or displacement
from a buffer of IMU measurements. We have found that a major limitation of
these methods is that their performance degrades heavily when applied to
motion patterns outside of the training domain (e.g., a network trained with a
walking robot would not generalize to a drone).

In most state estimation pipelines exteroceptive sensors, such as cameras or
lidars, are used to constrain and correct the drift of IMU integration.
Typically IMUs provide high frequency measurements (100 to 1000 Hz) while the
constraining measurements are at a lower rate (10 to 30 Hz). Learned inertial
velocity or displacement measurements can also be used as a low-frequency
constraining measurements[liu2020tlio]. The reason that IMU drift can be
constrained is therefore not because exteroceptive measurements are inherently
better, but rather because they enable a back-end estimation pipeline e.g. a
Kalman Filter or a smoother, to observe the IMU biases. Simply put, if IMU
biases could be observed directly from the IMU, exteroceptive sensing would
not be needed for odometry.

In this paper, we propose a new method which uses deep learning to predict IMU
biases directly. Our method trains a network to learn the process model of the
biases of a specific IMU. As a result, our method is sensor specific, not
platform specific and does not require periodic motion. The contributions in
this paper can be summarized as follows:

Figure 1: Proposed factor graph framework with learned predictions represented
as unary factors on the bias states. IMU pre-integrated factor from
[Forster2015] and visual factor from [Wisth_2021].

  * •

The first neural network capable of predicting the bias evolution of an IMU.

  * •

Integration of this network into a state estimation pipeline for improved
estimation in visually challenging scenarios.

  * •

Real-world experiments with handheld sensor payloads.

##  II RELATED WORK

In this section we summarize the growing field of inertial learning. In
Section II-A we discuss methods which learn the motion model of subjects while
Section II-B covers methods which learn a noise model of the IMU sensor.

###  II-A Learning Motion Models

There is strong demand for motion estimation algorithms which do not rely on
cameras. Most methods train a deep neural network to infer motion directly
from IMU data — typically focused on tracking a handheld mobile phone. These
methods have shown impressive performance compared to traditional inertial-
only pedestrian tracking methods.

The first data driven INS was IoNet Chen et al. [Chen2018], which inferred 2D
displacement and orientation change from buffered IMU data. Later, Yan et al.
[Yan2019] proposed RoNIN, a similar network which infers 2D velocity and
orientation from raw IMU data. TLIO, Liu et al. [liu2020tlio], proposed a
method to infer 3D motion and, in turn, to estimate high-fidelity trajectories
in a filtering framework. They use an Extended Kalman Filter (EKF) to estimate
the full 6 DoF state. Process updates were performed by traditional IMU
mechanization while measurement updates come from the network as relative
position measurements. This allowed the filter to estimate IMU biases.

In our earlier work we built upon the approach of [liu2020tlio] to develop a
more flexible system which could incorporate any additional sensing such as
cameras, lidar or legged robot kinematics [buchanan-corl]. However this
method, like the others discussed above, was based on learning a motion model.
These methods are susceptible to failure if applied outside of the training
domain (e.g., a network trained with pedestrian motion would not generalize to
a person on a bicycle) or when faced with unexpected motions such as slipping.

###  II-B Learning IMU Noise Models

The application of inertial learning to robotics is currently limited but the
existing methods tend to focus more on learning IMU noise models. Zhang et al.
[zhang2021imu] trained an RNN to denoise IMU measurements. Their network used
IMU measurements as input to predict a ”correct” measurement, reducing
integration error. This eliminated the need for the model to learn the
underlying motion equations and implies that the network learned the IMU’s
noise model. However, the network did not differentiate between bias and white
noise making the output measurements difficult to interpret.

Brossard et al. [Brossard20ral] used a Convolutional Neural Network (CNN) to
denoise a gyroscope — using the predictions to correct for the true angular
velocity. The CNN used dilated convolutions to account for temporal coverage
of the sequence data. Li et al. [Li21fira] built upon [Brossard20ral] with a
log-cosine loss function and multi-scale training regime to improve results.
However, neither method explicitly estimated biases and instead learned
uninterpretable corrections for each measurement thus limiting the ability to
fuse with other sensors.

##  III Problem Statement

Our aim is to estimate the position, orientation, linear velocity and IMU
biases of a mobile platform (in our experiments, we use a handheld sensor
payload) equipped with an IMU and either a lidar or stereo camera.

We illustrate the relevant reference frames in Fig. 2. This includes a
platform-fixed base frame $\mathtt{{B}}$, camera frame $\mathtt{C}$, IMU frame
$\mathtt{I}$, and lidar frame $\mathtt{L}$. The position
$\tensor[_{\mathtt{{W}}}]{\mathbf{p}}{{}_{\mathtt{{W}}\mathtt{{B}}}}$ and
orientation $\mathbf{R}_{\mathtt{{W}}\mathtt{{B}}}$ of the base are expressed
in world coordinates and can be jointly expressed with a homogeneous transform
$\tensor[_{\mathtt{{W}}}]{\mathbf{T}}{{}_{\mathtt{{W}}\mathtt{{B}}}}\in\mathrm{SE}(3)$.
Velocity $\tensor[_{\mathtt{{B}}}]{\mathbf{v}}{{}_{\mathtt{{W}}\mathtt{{B}}}}$
is expressed in the base frame and IMU biases
$\tensor[_{\mathtt{{I}}}]{\mathbf{b}}{{}^{g}},\;\tensor[_{\mathtt{{I}}}]{\mathbf{b}}{{}^{a}}$
are expressed in the IMU frame.

Figure 2: Reference frames for the two handheld devices. The world frame
$\mathtt{W}$ is a fixed frame, while the base frame $\mathtt{{B}}$, camera
frame $\mathtt{C}$, IMU frame, $\mathtt{I}$, and lidar frame $\mathtt{L}$ are
attached to the device. For simplicity, $\mathtt{L}$ and $\mathtt{{B}}$ are
coincident on the left handheld device and $\mathtt{C}$ and $\mathtt{{B}}$ on
the right.

###  III-A State Definition

The mobile platform state at time $t_{i}$ is defined as follows:

$\boldsymbol{x}_{i}\triangleq\left[\mathbf{R}_{i},\mathbf{p}_{i},\mathbf{v}_{i},\mathbf{b}^{g}_{i}\,\;\mathbf{b}^{a}_{i}\right]\in\mathrm{SO}(3)\times\mathbb{R}^{12}$
(1)

where: $\mathbf{R}_{i}$ is the orientation, $\mathbf{p}_{i}$ is the position,
$\mathbf{v}_{i}$ is the linear velocity, and the last two elements are the IMU
gyroscope and accelerometer biases.

##  IV IMU NOISE MODEL

A MEMS IMU typically contains six sensors, one for each dimension of an
accelerometer and a gyroscope which measure linear acceleration $\mathbf{a}$
in $\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}^{2}$ and rotational rate
$\boldsymbol{\omega}$ $\text{\,}\mathrm{rad}\mathrm{/}\mathrm{s}$
respectively. The true acceleration and rotation rate
$\\{\tilde{\mathbf{a}},\tilde{\boldsymbol{\omega}}\\}$ of a body expressed in
an Earth-Fixed coordinate frame $\mathtt{W}$ can be estimated using the IMU as
follows:

$\displaystyle\begin{split}\tilde{\mathbf{a}}(t)=&\mathbf{R}_{WI}^{T}(t)(\mathbf{a}(t)-_{w}g)+{\mathbf{b}}^{a}(t)+\boldsymbol{\eta}^{a}(t)\\\
\tilde{\boldsymbol{\omega}}(t)=&\boldsymbol{\omega}(t)+{\mathbf{b}}^{g}(t)+\boldsymbol{\eta}^{g}(t)\end{split}$
(2)

Where $R_{WI}$ is the rotation matrix from the world frame $\mathtt{W}$ to the
IMU frame $\mathtt{I}$ and ${}_{w}g$ is acceleration due to gravity expressed
in $\mathtt{W}$. The parameters $\boldsymbol{\eta}^{a}$ and
$\boldsymbol{\eta}^{g}$ represent the additive white noise present in all IMUs
and are modeled as zero-mean Gaussian random distributions.

The IMU biases ${\mathbf{b}}^{a}$ and ${\mathbf{b}}^{g}$ are physical
properties of the IMU and vary with time. Typically their evolution is modeled
as Brownian noise process, where the derivative of the biases are sampled from
zero-mean Gaussian distributions:

$\displaystyle\begin{split}\dot{{\mathbf{b}}^{a}}(t)&=\boldsymbol{\eta}^{ba}\\\
\dot{{\mathbf{b}}^{g}}(t)&=\boldsymbol{\eta}^{bg}\end{split}$ (3)

This can be re-written in discrete time by integrating between two time steps
$[t_{i},t_{j}]$:

$\displaystyle\begin{split}{\mathbf{b}}^{a}_{j}&={\mathbf{b}}^{a}_{i}+\boldsymbol{\eta}^{bad}\\\
{\mathbf{b}}^{g}_{i}&={\mathbf{b}}^{g}_{i}+\boldsymbol{\eta}^{bgd}\end{split}$
(4)

Where $\boldsymbol{\eta}^{bad}$ is related to $\boldsymbol{\eta}^{ba}$ by the
IMU sampling time:
$\boldsymbol{\eta}^{bad}=\boldsymbol{\eta}^{ba}/\sqrt{(}\Delta t)$.

##  V LEARNING THE BIAS PROCESS MODEL

While the IMU bias process is often modeled as Brownian noise, this does not
hold for longer periods of time. The true underlying dynamics are in fact a
highly non-linear function which depends on the IMU vibration, impact and
temperature [strapdown-ins]. This problem thus lends itself to deep learning —
which can learn highly non-linear functions. To this end, rather than employ
(4), we propose a deep neural network ${\cal F}$ with parameters $\theta$ such
that

${\cal F}_{\theta}:\left({\cal
I}_{ij},{\mathbf{b}}^{a}_{i},{\mathbf{b}}^{g}_{i}\right)\mapsto(\hat{{\mathbf{b}}^{a}_{j}},\hat{{\mathbf{b}}^{g}_{j}})$
(5)

where ${\cal I}_{ij}$ is a buffer of IMU measurements between times $i$ and
$j$. The network outputs
$\left(\hat{{\mathbf{b}}^{a}_{j}},\hat{{\mathbf{b}}^{g}_{j}}\right)$ are
predictions of the next IMU bias value at time $j$.

###  V-A Network Architecture

We implemented two different network implementations to approximate (5). The
first, is a standard autoregressive Long Short-Term Memory (LSTM) which we
refer to as LSTM [hochreiter1997long] (see Fig. 3). The second is very similar
but it also takes as input the previous bias estimate, either from the network
or from factor graph optimizer and we refer to this as LSTM-P. Although very
similar, the difference is critical because while LSTM-P assumes an initial
IMU bias has been estimated, possibly leaving the IMU rest for several
seconds, LSTM does not. This means LSTM would be used for in-motion IMU bias
initialization as it does not require an initial guess. LSTM-P was trained
using teacher forcing [teacherForcing].

Figure 3: LSTM architecture. 1 s buffers of IMU data are passed to the LSTM
which outputs a prediction of the bias. As a recurrent network, the LSTM is
capable of learning the IMU process over a history of buffers.

###  V-B Loss Function

The network predicts the bias values directly and so we simply use the Mean
Square Error (MSE) as a loss function:

${\cal
L}({\mathbf{b}}^{a},\hat{{\mathbf{b}}^{a}})=\frac{1}{n}\sum^{n}_{k=1}\left\lVert{\mathbf{b}}^{a}_{k}-\hat{{\mathbf{b}}^{a}}_{k}\right\rVert^{2},$
(6)

We use the same approach for gyroscope bias loss ${\cal
L}({\mathbf{b}}^{g},\hat{{\mathbf{b}}^{g}})$ which we learn separately from
the accelerometer.

###  V-C Datasets

For training and experiments we used two datasets with two different sensor
payloads as shown in Table I. The first dataset is the Newer College Dataset
(NCD) [ramezani2020newer] which uses an Ouster OS1 lidar with integrated IMU.
The second dataset is the Newer College Multi-Camera Dataset (NCD-
Multi)[zhang2021multicamera] which uses an Alphasense multi-camera sensor,
although in this work we only use the front facing stereo cameras.

Ground-truth was generated using the on-board lidars to localize in a prior
high-quality map of the environment. We build a factor graph with nodes for
every lidar measurement (10 Hz), unary lidar localization factors on each node
and IMU factors between each node. This allows us to generate ground-truth IMU
biases at 10Hz which we then linearly interpolate to 100 Hz for training.

TABLE I: Sensors Used for Experiments.

Dataset | Exteroception | IMU | Experiment
---|---|---|---
Newer College Dataset (NCD) | Lidar | ICM-20948 | Section VI-A
NCD Multi-Cam (NCD-Multi) | Stereo Vision | BMI085 | Section VI-B

###  V-D Training

We trained two models, one using the LSTM architecture for NCD and one with
LSTM-Teacher for NCD-Multi. In each case a sequence was taken from each
dataset to be used for later testing. The remaining sequences were split 75:25
into training and validation datasets.

We used the Adam optimizer with a learning rate of $10^{-4}$. Training lasted
200 epochs with the model minimizing validation error used. Training takes
approximately 1 hour on a laptop with an Nvidia Quadro P2000 GPU.

Figure 4: Results of the in-motion initialization experiment with the NCD, see
Section VI-A for more details. Vilens Static is initialized at the beginning,
Vilens In-Motion is initialized while in motion and Learned is also
initialized in-motion but uses learned bias predictions.

###  V-E Factor Graphs

Bias predictions can be incorporated into a factor graph-based state estimator
as unary factors on the bias estimate, as in Fig. 1. The resulting residuals
$\mathbf{r}_{{\mathbf{b}}^{a}}$ and $\mathbf{r}_{{\mathbf{b}}^{g}}$ are then:

$\begin{split}\mathbf{r}_{{\mathbf{b}}^{a}}=\left\lVert{\mathbf{b}}^{a}_{j}-\hat{{\mathbf{b}}^{a}_{j}}\right\rVert^{2}_{\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{a}}}\\\
\mathbf{r}_{{\mathbf{b}}^{g}}=\left\lVert{\mathbf{b}}^{g}_{j}-\hat{{\mathbf{b}}^{g}_{j}}\right\rVert^{2}_{\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{g}}}\end{split}$
(7)

There has been significant research on training predictive neural networks to
provide uncertainty estimates associated with a prediction [liu2020tlio,
buchanan-corl]. However, for now, we hand-tune the covariances
$\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{a}}$ and
$\hat{\mathbf{\Sigma}}_{{\mathbf{b}}^{g}}$ to fixed values.

##  VI EXPERIMENTS AND RESULTS

In this section we describe the results of training the neural network, as
well as the state estimation experiments which incorporate the bias
predictions.

###  VI-A Initialization while in Motion

The LSTM network is capable of providing IMU bias predictions without first
being provided any initial bias. This ability allows the network to be used to
bootstrap a state estimation pipeline, enabling in-motion initialization. This
could be a significantly useful for analyzing datasets recorded by non-
experts, such as industrial workers or everyday people, who may not know to
place a sensor payload on a flat surface for several seconds to initialize the
biases.

In Fig. 4 we show the results of a sequence from NCD which was started while
the person was in-motion. The baseline is the Vilens states estimation
algorithm [Wisth_2021] which uses ICP to create relative pose factors between
nodes in the factor graph. In the first case, the algorithm is allowed to
start from the beginning of the sequence when the IMU is at rest so that the
biases can be initialized, this is referred to as Vilens Static Initialization
in Fig. 4. Vilens In-Motion Initialization is the case where state estimation
is started while the IMU is moving. As can be seen from the figure, the biases
take an extremely long time to converge to the correct value. When adding bias
predictions, the biases almost immediately converge, reducing trajectory
error.

Figure 5: Visual challenges in the NCD-Multi Stairs sequence used in the
experiment in Section VI-B. Left: Door opening obscuring the sensors, Right:
Tight spaces about 1.5 m wide and a mirror.

###  VI-B Odometry in Visually Challenging Scenarios

There exist many common situations or environments in which visual odometry
systems degrade or even fully fail. Conditions such as door-opening, narrow
spaces and fast rotations are common challenges. We show in Fig. 6 that the
bias predictions can improve visual inertial state estimation in these
conditions. Our proposed approach improves upon pure IMU-only mechanization in
periods where visual feature tracking fails.

The Stairs sequence from NCD-Multi is particularly challenging as shown in
Fig. 5. As such the stereo visual odometry struggles and loses track of all
features several times during the sequence. By adding bias prediction to the
factor graph as in Fig 1, the mean Relative Pose Error (RPE) over 10 m was
reduced from 0.52 m to 0.29 m (56% reduction).

Figure 6: Results of the Stairs sequence from NCD-Multi, see Section VI-B for
further details. Vilens is baseline visual-inertial odometry using
[Wisth_2021] and Learned Biases is the proposed factor graph in Fig. 1.

##  VII CONCLUSION

We present a novel application of machine learning to inertial navigation. Our
method is not robot specific and does not require periodic locomotion. The
learned bias predictions can improve state estimation in visually challenging
situations or enable in-motion initialization. Future work will further
explore direct comparisons between existing IMU learning methods and perform
additional experiments.