# DIDO: Deep Inertial Quadrotor Dynamical Odometry

Kunyi Zhang1,2, Chenxing Jiang1,2, Jinghang Li2, Sheng Yang3, Teng Ma3, Chao
Xu1,2, and Fei Gao1,2 Manuscript received: February, 24, 2022; Revised: May,
19, 2022; Accepted: June, 17, 2022. This paper was recommended for publication
by Editor Sven Behnke upon evaluation of the Associate Editor and Reviewers’
comments. This work was supported by the National Key Research and Development
Program of China (Grant NO. 2020AAA0108104), Alibaba Innovative Research (AIR)
Program, and National natural Science Foundation of China under Grant
62003299. (Corresponding author: Fei Gao.)1State Key Laboratory of Industrial
Control Technology, Institute of Cyber-Systems and Control, Zhejiang
University, Hangzhou 310027, China.2Huzhou Institute, Zhejiang University,
Huzhou 313000, China.3Alibaba DAMO Academy Autonomous Driving Lab, Hangzhou
311121, China.E-mail: {kunyizhang<EMAIL_ADDRESS>Object Identifier
(DOI): see top of this page.

###### Abstract

In this work, we propose an interoceptive-only state estimation system for a
quadrotor with deep neural network processing, where the quadrotor dynamics is
considered as a perceptive supplement of the inertial kinematics. To improve
the precision of multi-sensor fusion, we train cascaded networks on real-world
quadrotor flight data to learn IMU kinematic properties, quadrotor dynamic
characteristics, and motion states of the quadrotor along with their
uncertainty information, respectively. This encoded information empowers us to
address the issues of IMU bias stability, quadrotor dynamics, and multi-sensor
calibration during sensor fusion. The above multi-source information is fused
into a two-stage Extended Kalman Filter (EKF) framework for better estimation.
Experiments have demonstrated the advantages of our proposed work over several
conventional and learning-based methods.

###### Index Terms:

Localization; Visual-Inertial SLAM; Sensorimotor Learning

##  I Introduction

Aerial robots are popular autonomous vehicles prevalently used in
entertainment and industrial applications. To locate themselves, exteroceptive
sensors (e.g., LiDAR and cameras) play a prominent role in common scenarios.
However, when encountering degenerate cases where features are either heavily
repeated or insufficient, these robots in-turn rely on interoceptive sensors
(e.g., IMU and rotor tachometer) to deduct their poses. Inaccurate dynamics
models, unestimated bias can lead to rapid divergence of state propagation.

Recent approaches [1, 2, 3, 4, 5, 6], for exteroceptive sensor disabled
scenarios, choose to learn the correspondence between IMU sequences and the
relative poses through a data-driven approach. Although these works show the
potential of deep neural networks for estimation, they are typically designed
for scenes of walking pedestrians or low-degree-of-freedom (DOF) ground
vehicles, which is difficult to generalize to aerial robots.

As a 4 DOF under-actuated system, there is a coupling of kinematic states
between the velocity and tilt of the quadrotor, which are not fully exploited
in the general state estimation system. The algorithm proposed in [7] can
obtain the tilt and velocity of a quadrotor using only the tachometer and IMU
data. However, its dynamics model only considers a quadratic thrust model and
a first-order drag model, which the quadrotors do not strictly conform to.
NeuroBEM [8] employs a neural network to compensate for the unmodeled parts
such as airflow during the dynamics modeling, but it is not utilized for state
estimation.

Figure 1:  The filter pipeline of proposed Deep Inertial quadrotor Dynamical
Odometry (DIDO). The green parts are cascaded networks designed for learning
kinematic properties, quadrotor dynamic characteristics, and motion states.
The blue parts are EKF processes.

Therefore, this paper proposes a deep inertial-dynamical odometry framework,
employing neural networks to learn both IMU and dynamics properties for better
performance in quadrotor state estimation. The kinematic properties of IMU
(accelerometer and gyroscope) are learned by two convolutional neural networks
(CNN) for measurement de-biasing. The dynamic characteristics of quadrotors,
with respect to the propeller rotation speed and states, are learned to
compensate for mechanistic modeling by a CNN. Besides, the velocity and
position of the quadrotor are observed in real-time by a recurrent neural
network (RNN). Finally, the networks mentioned above are fused with IMU and
tachometer measurements in a two-stage EKF framework. Specifically, in the
stage of rotation, the de-biased angular velocity and acceleration are fused
to get accurate rotation by gravity-alignment. In the translation stage, the
modified dynamics is regard as process model, and the acceleration along with
velocity and position from the network are considered as observation models,
for accurate estimation of kinematic, dynamic states, and extrinsic
parameters. The letter’s contributions are enumerated below:

  1. 1.

We present a series of networks to learn the IMU kinematics and quadrotor
dynamics, demonstrating the capability to provide precise observations with
only raw IMU and tachometer data.

  2. 2.

We propose a complete state estimation system that combines quadrotor dynamics
and network observations within a two-stage EKF to jointly estimate kinematic,
dynamic states, and extrinsic parameters.

##  II Related Works

###  II-A IMU Learning based estimation

IONet [1], RoNIN [2] and RIDI [3] are end-to-end learning methods for pose
estimation. IONet [1] uses a long short-term memory network (LSTM) based
network, whose inputs are accelerometer and gyroscope measurements within a
time window in the world coordinate system. RoNIN [2] proposes three different
neural network architectures to solve the inertial navigation problem: LSTM,
temporal convolutional network (TCN), and residual network (ResNet). These
models regress the velocity and displacement of the pedestrian in 2
dimensions. RIDI [3] design a two-stage system, which first regresses the
velocity of pedestrians and then optimizes correction terms for IMU to match
the velocity. The displacement of a pedestrian is finally obtained after
integration.

AI-IMU [4], IDOL [5], TLIO [6] and CTIN [9] combines deep neural networks with
filter framework to obtain better performance. AI-IMU [4] is specially
designed for the purpose of vehicle’s pose estimation, using a CNN to estimate
the covariance of accelerometer and gyroscope measurements and applying them
to an invariant extended Kalman filter (IEKF). In addition, the filter
considers the vehicle sideslip constraints as well as the motion constraints
in the vertical direction. IDOL [5] and TLIO [6] are both applied for
pedestrian’s pose estimation. IDOL [5] is divided into a rotation estimation
module and a translation estimation module. In the rotation module, an LSTM
network with gyroscope, accelerometer, and magnetometer data as inputs is
trained to output the rotation, which is fused with the rotation obtained by
integrating the gyroscope using the extended Kalman filter (EKF). In the
position estimation module, the position is regressed using the LSTM network
after converting the acceleration to the world coordinate system using the
above rotation. TLIO [6] uses a ResNet network similar to RoNIN [2] to
estimate pedestrian displacements without yaw, and a stochastic cloning
extended Kalman filter (SCEKF) [10] framework to fuse the output values of the
network with raw IMU measurements, for estimating position, rotation, and
sensor bias in a tightly coupled manner. CTIN [9] uses ResNet and LSTM to
obtain the local and global embeddings, and feeds these two embeddings into
two multi-headed Transformers [11] to obtain the velocity and its covariance.

###  II-B Quadrotor dynamics and state estimation

A quadrotor is usually powered by four rotating propellers, and the
relationship between rotors’ speed and dynamical states can be obtained by
modeling the quadrotor’s mechanics. Traditionally, a simple quadratic model is
widely adopted, where the thrust and axial torque generated by the rotor are
proportional to the square of their rotation speed with a constant factor
[12]. The momentum theorem does not take into account either the motion of the
rotor in air or the interactions between these rotors and the body. The blade-
element-momentum (BEM) theory [13, 14, 15] combines blade-element theory and
momentum theory to alleviate the difficulty of calculating the induced
velocity and accurately capture the aerodynamic force and torque acting on
single rotor in a wide range of operating conditions. But it also does not
take into account any interactions between different propellers and body.

NeuroBEM [8] uses a neural network to fit the residuals between the real
quadrotor dynamics and the calculated values from the BEM theory. It [8] takes
the rotor speed and kinematic states as the network’s inputs to allow the
quadrotor to perform highly maneuverable trajectories. In [7], the authors
estimate the tilt and velocity of the quadrotor in the body frame by applying
only the tachometer and gyroscope as inputs and taking the accelerometer as
observation. VIMO [16] and VID-Fusion [17] also consider the constraints of
quadrotor dynamics in visual inertial fusion to improve the robustness and
accuracy of pose estimation.

##  III Kinematic and Dynamic Network Design

###  III-A Preliminaries

####  III-A1 IMU Model

IMU measurements include the gyroscope $\widetilde{\bm{\omega}}$ and non-
gravitational acceleration $\widetilde{\bm{a}}$, which are measured in the IMU
frame (the $\mathcal{I}$ frame with IMU sensor as the center, Front-Left-Up as
the order of three axes) and given by:

$\displaystyle{{}^{\mathcal{I}}}\widetilde{\bm{\omega}}$
$\displaystyle={{}^{\mathcal{I}}}{\bm{\omega}}+{{}^{\mathcal{I}}}{\bm{b}_{\bm{\omega}}}+{{\bm{n}}_{\bm{\omega}}},$
(1) $\displaystyle{{}^{\mathcal{I}}}\widetilde{\bm{a}}$
$\displaystyle={{}^{\mathcal{I}}}{\bm{a}}+{{}^{\mathcal{I}}}{\bm{b}_{\bm{a}}}+{{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}\bm{\mathrm{g}}+{{\bm{n}}_{\bm{a}}},$

where ${{}^{\mathcal{I}}}{\bm{\omega}}$ and ${{}^{\mathcal{I}}}{\bm{a}}$ are
the true angular velocity and acceleration,
${{}^{\mathcal{G}}}\bm{\mathrm{g}}=[0,0,9.8]$ (unit: $m/s^{2}$) is the gravity
vector in the gravity-aligned frame (the ${\mathcal{G}}$ frame with z-axis
pointing down vertically), ${{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}$ is the
rotation matrix from the $\mathcal{I}$ frame to the ${\mathcal{G}}$ frame,
${\bm{n}}_{\bm{\omega}}$ and ${{\bm{n}}_{\bm{a}}}$ are the additive Gaussian
white noise in gyroscope and acceleration measurements,
${\bm{b}_{\bm{\omega}}}$ and ${\bm{b}_{\bm{a}}}$ are the bias of IMU modeled
as random walk:

$\displaystyle{{\bm{n}}_{\bm{\omega}}}$
$\displaystyle\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{\omega}}^{2}),$
$\displaystyle\dot{\bm{b}}_{\bm{\omega}}$
$\displaystyle\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{b}_{\bm{\omega}}}^{2}),$ (2)
$\displaystyle{{\bm{n}}_{\bm{a}}}$
$\displaystyle\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{a}}^{2}),$
$\displaystyle\dot{\bm{b}}_{\bm{a}}$
$\displaystyle\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{b}_{\bm{a}}}^{2}).$

####  III-A2 Quadrotor Dynamics

Because the propagation of kinematic states is driven by multiple propulsion
units in a quadrotor system, we model the Newtonian dynamics according to
[18]. The total driving force of a quadrotor in the body frame (the
$\mathcal{B}$ frame with center of mass as the center, the same order as the
$\mathcal{I}$ frame) is the sum of the thrust ${{}^{\mathcal{B}}\bm{F}_{t}}$
and drag force ${{}^{\mathcal{B}}\bm{F}_{d}}$ generated by each propulsion
unit as follow:

$\displaystyle{}^{\mathcal{B}}\bm{F}=\sum_{i=1}^{4}\left({{}^{\mathcal{B}}\bm{F}}_{t_{i}}-{{}^{\mathcal{B}}\bm{F}}_{d_{i}}\right)=\sum_{i=1}^{4}\left(\tau
u_{i}^{2}\bm{e}_{3}-u_{i}D{{}^{\mathcal{B}}\bm{v}}_{i}\right),$ (3)

where $\tau$ is the thrust coefficient for the propellers,
$D=diag(d_{x},d_{y},d_{z})$ is the matrix of effective linear drag
coefficients, $\bm{e}_{3}=[0,0,1]^{\top}$ is the $z$ axis in any frame, and
$u_{i}$ and ${{}^{\mathcal{B}}\bm{v}}_{i}$ are the rotation speed and velocity
of the $i$-th rotor, respectively. Actually, the velocity of each rotor is
${{}^{\mathcal{B}}}\bm{v}_{i}={{}^{\mathcal{B}}}\bm{v}+{{}^{\mathcal{B}}}{\bm{\omega}}\times{{}^{\mathcal{B}}}\bm{r}{{}_{i}^{\mathcal{B}}}$,
where ${{}^{\mathcal{B}}}\bm{v}$ and ${{}^{\mathcal{B}}}{\bm{\omega}}$ are the
linear and angular velocity of the quadrotor’s center of mass (COM),
${{}^{\mathcal{B}}}\bm{r}{{}_{i}^{\mathcal{B}}}$ is the position of the $i$-th
rotor relative to the COM. To simplify the calculation, we ignore the velocity
discrepancy of different rotors and express it as
${{}^{\mathcal{B}}}\bm{v}_{i}\approx{{}^{\mathcal{B}}}\bm{v}$. The input
notations are abbreviated as $U_{ss}=\sum_{i=1}^{4}u_{i}^{2}$ and
$U_{s}=\sum_{i=1}^{4}u_{i}$. Therefore, we can obtain the Newtonian equation
in the ${\mathcal{G}}$ frame:

$\displaystyle m\frac{d}{dt}\left({{{}^{\mathcal{G}}}\bm{v}}\right)$
$\displaystyle={{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}\left(\tau
U_{ss}{\bm{e}}_{3}-U_{s}D{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{{}^{\mathcal{G}}}\bm{v}{{}_{\mathcal{B}}^{\mathcal{G}}}}\right)-m{{}^{\mathcal{G}}}\bm{\mathrm{g}}.$
(4)

Note that, we divide the rotor speed by $10,000$ to ensure the numerical
stability.

###  III-B Network Design

####  III-B1 De-Bias Net

As an interoceptive sensor, IMU often requires several integrations to obtain
the kinematic states, but the system would drift or diverge because of the
failure to accurately estimate the accelerometer bias
${{}^{\mathcal{I}}}\bm{b}_{a}$ and gyroscope bias
${{}^{\mathcal{I}}}\bm{b}_{\omega}$. So, we design a De-Bias Net to learn the
kinematic characteristics of IMU for de-biasing. The two De-Bias Nets for
accelerometer and gyroscope have the same network architecture, a 1D version
of ResNet [19] with only one residual block to boost the inference process.
Fully-connected layers are extended to output.

The respective input features of both De-Bias Nets are historical raw
accelerometer measurements ${{}^{\mathcal{I}}}\widetilde{\bm{a}}$ and
gyroscope measurements ${{}^{\mathcal{I}}}\widetilde{\bm{\omega}}$, each
output is ${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{a}}$ and
${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{\omega}}$ at every moment,
separately.

We define two Mean Square Error (MSE) loss functions
$\mathcal{L}_{\text{MSE},{\bm{a}}}$ and
$\mathcal{L}_{\text{MSE},{\bm{\omega}}}$ for accelerometer and gyroscope,
respectively, on the following integrated increments:

$\displaystyle\mathcal{L}_{\text{MSE},{\bm{a}}}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}{\rVert{\bm{v}}_{i,i+n}-{\bm{\widehat{v}}}_{i,i+n}\rVert^{2}_{2}},$
(5) $\displaystyle\mathcal{L}_{\text{MSE},{\bm{\omega}}}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}{\|\log\left(\left({\bm{\widehat{q}}}_{i,i+n}\right)^{*}\otimes{\bm{q}}_{i,i+n}\right)\|^{2}_{2}},$

where

$\displaystyle{\bm{\widehat{v}}}_{i,i+n}$
$\displaystyle=\int_{i}^{i+n}{{}_{\mathcal{I}_{t}}^{\mathcal{G}}}\mathbf{R}({{}^{\mathcal{I}_{t}}}\widetilde{\bm{a}}-{{}^{\mathcal{I}_{t}}}\widehat{\bm{b}}_{\bm{a}})dt,$
(6) $\displaystyle{\bm{v}}_{i,i+n}$
$\displaystyle={{}^{\mathcal{G}}}\bm{v}_{i+n}-{{}^{\mathcal{G}}}\bm{v}_{i},$
$\displaystyle{\bm{\widehat{q}}}_{i,i+n}$
$\displaystyle=\int_{i}^{i+n}\frac{1}{2}{{}_{\mathcal{I}_{t}}^{\mathcal{I}_{i}}}{\bm{q}}\otimes({{}^{\mathcal{I}_{t}}}\widetilde{\bm{\omega}}-{{}^{\mathcal{I}_{t}}}\widehat{\bm{b}}_{\bm{\omega}})dt,$
$\displaystyle{{\bm{q}}}_{i,i+n}$
$\displaystyle=({{}_{\mathcal{I}_{i}}^{\mathcal{G}}}{\bm{q}})^{*}\otimes{{}_{\mathcal{I}_{i+n}}^{\mathcal{G}}}{\bm{q}}.$

Note that, the subscript $i$ is the abbreviation for time $t_{i}$,
${\bm{\widehat{v}}}_{i,i+n}$ and ${\bm{\widehat{q}}}_{i,i+n}$ are the
intermediate variables calculated from the estimated values
${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{a}}$ and
${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{\omega}}$ of the De-Bias Net by the
forward Euler method actually, ${\bm{v}}_{i,i+n}$ and ${{\bm{q}}}_{i,i+n}$ are
the ground truth velocity increment and relative rotation of each sliding
window, $n$ is the window size, $N$ is the batch size, and $\otimes$ means the
quaternion multiplication. In training,
${{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}$ is the ground truth rotation.

####  III-B2 Res-Dynamics Net

Actually, the dynamics model of quadratic thrust and approximately linear drag
in Eq. (3) can not accurately model quadrotors because of complicated effects
related to airflow and so on. Therefore, like NeuroBEM [8], we adopt the same
ResNet [19] as De-Bias Nets to capture the unmodeled part of the quadrotor
dynamics.

We regard a sliding window of angular and linear velocity in the $\mathcal{B}$
and rotor speed data as input features to train the Res-Dynamics Net in a
supervised fashion, whose history window size is set to 20.

In order to fit the unmodeled force and fuse it into the EKF framework
reasonably, a MSE loss function is replaced by a Negative log-likelihood (NLL)
loss [20] after the MSE loss stabilizes and converges:

$\displaystyle\mathcal{L}_{\text{MSE},\bm{f}}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}{\rVert{{}^{\mathcal{B}}}{\bm{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}_{i}-{{}^{\mathcal{B}}}\bm{\widehat{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}_{i}\rVert^{2}_{2}},$
(7) $\displaystyle\mathcal{L}_{\text{NLL},\bm{f}}$
$\displaystyle=\frac{1}{2N}\sum_{i=1}^{N}\log\det(\widehat{\bm{\Sigma}}^{2}_{\bm{f}_{i}})$
$\displaystyle+\frac{1}{2N}\sum_{i=1}^{N}{\rVert{{}^{\mathcal{B}}}{\bm{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}_{i}-{{}^{\mathcal{B}}}\bm{\widehat{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}_{i}\rVert^{2}_{\widehat{\bm{\Sigma}}^{2}_{\bm{f}_{i}}}},$

where

$\displaystyle{{}^{\mathcal{B}}}\bm{\widehat{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}$
$\displaystyle=\frac{1}{m}(\tau
U_{ss}{\bm{e}}_{3}-U_{s}D{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{{}^{\mathcal{G}}}\bm{v}{{}_{\mathcal{B}}^{\mathcal{G}}}}+\bm{\widehat{f}}_{res})-{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}\bm{\mathrm{g}},$
(8) $\displaystyle{{}^{\mathcal{B}}}{\bm{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}$
$\displaystyle={{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}{\bm{a}}{{}^{\mathcal{G}}_{\mathcal{I}}}+{{}_{\mathcal{B}}^{\mathcal{I}}}\mathbf{R}^{\top}({{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}\times({{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}\times{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}})+{{}^{\mathcal{I}}}\bm{{\bm{\widehat{\alpha}}}}\times{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}).$

Note that, $\bm{\widehat{f}}_{res}$ and $\widehat{\bm{\Sigma}}^{2}_{\bm{f}}$
are the unmodeled force and its corresponding covariance,
${{}^{\mathcal{B}}}\bm{\widehat{a}}{{}^{\mathcal{G}}_{\mathcal{B}}}$ is the
intermediate variable calculated from $\bm{\widehat{f}}_{res}$,
${{}^{\mathcal{G}}}{\bm{a}}{{}^{\mathcal{G}}_{\mathcal{I}}}$ is the ground
truth of acceleration,
${{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}={{}^{\mathcal{I}}}{\widetilde{\bm{\omega}}}-{{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{\omega}}$,
${{}^{\mathcal{I}}}{\bm{\widehat{\alpha}}}=\frac{d}{dt}{{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}$
and ${{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}$ is the
extrinsic translation between the $\mathcal{I}$ and $\mathcal{B}$ frame. In
practice, we low-pass filter the
${\widehat{\bm{\omega}}},{\bm{\widehat{\alpha}}}$ to reduce noise.

Figure 2: The network structure for V-P Net.

####  III-B3 V-P Net

Even though De-Bias Nets help to reduce the bias of IMU measurements, the
remaining tiny offset of acceleration would not avoid the cumulative error in
the integration process. Therefore, we design a cascaded GRU network to learn
velocity and position approximation that has less error accumulation over a
long period of time. As every dimension of velocity and position are
independent to each other, we adopt three cascaded GRU networks for the $X$,
$Y$, and $Z$ axes separately. The cascaded architecture has two parts: V Net
and P Net, these two parts have the same structure as shown in Fig. 2.

Firstly, we integrate the de-biased acceleration as follow:

$\displaystyle\widehat{\bm{v}}_{k,k+m}$
$\displaystyle=\int_{k}^{k+m}{{}_{\mathcal{I}_{t}}^{\mathcal{G}}}\mathbf{R}({{}^{\mathcal{I}_{t}}}\widetilde{\bm{a}}-{{}^{\mathcal{I}_{t}}}\widehat{\bm{b}}_{\bm{a}})dt,$
(9)

where the size of an integration window is $m$ without overlap among each
window. Each axis of $\bm{\widehat{v}}_{k,k+m}$ and integration time $dt$ are
input features of V Net. The hidden state is passed to fully-connected layers
to regress the uniaxial velocity relative to the start of a sequence.

Then, we use results of the V Net to calculate the displacement over the
integration window as shown in Eq. (10), which is the input feature of P Net
to regress the uniaxial position relative to the start of a sequence.

$\displaystyle\bm{\widehat{p}}_{k,k+m}=\bm{\widehat{v}}_{k+m}\bm{t}_{k,k+m}-\iint_{k}^{k+m}{{}_{\mathcal{I}_{t}}^{\mathcal{G}}}\mathbf{R}({{}^{\mathcal{I}_{t}}}\widetilde{\bm{a}}-{{}^{\mathcal{I}_{t}}}\widehat{\bm{b}}_{\bm{a}})dt^{2},$
(10)

where $\bm{\widehat{v}}_{k+m}=\bm{\widehat{v}}_{i,k+m}+\bm{\widehat{v}}_{i}$,
$\bm{\widehat{v}}_{i}$ is the initial velocity of the sequence,
$\bm{\widehat{v}}_{i,k+m}$ is the result of the V Net in the $(k+m)$-th step
and $\bm{t}_{k,k+m}$ is the duration of the integration window.

Similarly, we use MSE and NLL loss over the whole sequence as follows:

$\displaystyle\mathcal{L}_{\text{MSE},\bm{v},\bm{p}}$
$\displaystyle=\frac{1}{2NM}\sum_{i=0}^{N}\sum_{j=0}^{M}\left(\rVert\bm{v}_{i,j}-\bm{\widehat{v}}_{i,j}\rVert^{2}_{2}+\rVert\bm{p}_{i,j}-\bm{\widehat{p}}_{i,j}\rVert^{2}_{2}\right),$
(11) $\displaystyle\mathcal{L}_{\text{NLL},\bm{v},\bm{p}}$
$\displaystyle=\frac{1}{2NM}\sum_{i=1}^{N}\sum_{j=0}^{M}(\log\det(\widehat{\bm{\Sigma}}^{2}_{\bm{v}_{i,j}})+\log\det(\widehat{\bm{\Sigma}}^{2}_{\bm{p}_{i,j}}))$
$\displaystyle+\frac{1}{2NM}$
$\displaystyle\sum_{i=0}^{N}\sum_{j=0}^{M}\big{(}\rVert\bm{v}_{i,j}-\bm{\widehat{v}}_{i,j}\rVert^{2}_{\widehat{\bm{\Sigma}}^{2}_{\bm{v}_{i,j}}}+\rVert\bm{p}_{i,j}-\bm{\widehat{p}}_{i,j}\rVert^{2}_{\widehat{\bm{\Sigma}}^{2}_{\bm{p}_{i,j}}}\big{)},$

where
$\bm{v}_{i,j}={{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{I}_{j}}}-{{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{I}_{i}}}$,
$\bm{p}_{i,j}={{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{I}_{j}}}-{{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{I}_{i}}}$,
$M$ is the sequence length, $N$ is batch size, $\bm{\widehat{v}}_{i,j}$ and
$\bm{\widehat{p}}_{i,j}$ are the $j$-th predicted uniaxial velocity and
position relative to the start of sequence $i$, $\bm{v}_{i,j}$ and
$\bm{p}_{i,j}$ are the ground truth,
${\widehat{\bm{\Sigma}}^{2}_{\bm{v}_{i,j}}}$ and
${\widehat{\bm{\Sigma}}^{2}_{\bm{p}_{i,j}}}$ are the covariance of uniaxial
velocity and position.

###  III-C Dataset Preparation

There are only the Blackbird dataset [21] and the VID dataset [22] in the
flying robot community that contain quadrotor dynamics. The former
trajectories are too repetitive to generalize well for networks, while the
latter suffers from insufficient data for training. Therefore, to satisfy the
training requirements, we equip a new quadrotor as a data acquisition platform
and record adequate data. The quadrotor assembles 4 tachometers and an
additional IMU111https://www.xsens.com/mti-300 as shown in Fig. 3, and the
ground truth is provided by a VICON222https://www.vicon.com/ system.

To ensure the diversity of trajectories, we record a total of 274 yaw-constant
and yaw-forward trajectories (total time: $325~{}min$, total distance:
$19.92~{}km$), which include 247 Random, 15 Circle and 12 Figure-8
trajectories. Where Random trajectories are sampled and optimized [23] to be
executed by the quadrotor as a smooth trajectory.

To provide high-frequency and accurate supervised data for network training,
we first fit the full state of kinematics at the IMU frequency with B-splines
[24] in the gravity-aligned coordinate frame, and then identify the dynamics
model of the quadrotor by offline optimization.

Figure 3: The quadrotor platform, which assembles an IMU and 4 tachometers, is
built for data acquisition.

###  III-D Training Details

####  III-D1 Covariance Training

Except for De-Bias Net, the mean and covariance outputs of the corresponding
networks are available for the remaining three networks. Every covariance
$\widehat{\bm{\Sigma}}^{2}$ is a 3$\times$3 matrix with 6 degrees of freedom,
but a diagonal form is assumed in this paper and parameterized by 3
coefficients written as
$\widehat{\bm{\Sigma}}^{2}_{\bm{\xi}}=\text{diag}(e^{2\widehat{\bm{\xi}}_{x}},e^{2\widehat{\bm{\xi}}_{y}},e^{2\widehat{\bm{\xi}}_{z}}).$
Since it is difficult to converge the direct training covariance by
$\mathcal{L}_{\text{NLL}}$, we use $\mathcal{L}_{\text{MSE}}$ in the first 20
epoch and then replace by $\mathcal{L}_{\text{NLL}}$.

####  III-D2 V-P Net

When training V-P Net, we use different lengths of sequences. In experiment,
the hidden sizes of GRU are [2, 64, 128, 256], integration window size $m$ is
20.

####  III-D3 Data Separation and Optimizer

We select 200 Random trajectories for training and the remaining as the
validation and test sets. Adam optimizer is chose to minimize the loss
function, and the initial learning rate is $10^{-4}$. The length of validation
and test set is from 58.5 ($m$) to 107.1 ($m$).

|  ${{}^{\mathcal{G}}_{\mathcal{I}}}\bm{q}$ |  ${{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{B}}}$ |  ${{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}$ |  $\tau$ |  $\bm{d}$ |  ${{}^{\mathcal{I}}_{\mathcal{B}}}\bm{q}$ |  ${{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}$
---|---|---|---|---|---|---|---
value | ${{}^{\mathcal{G}}_{\mathcal{I}_{0}}}\bm{q}$ | ${{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{I}_{0}}}$ | ${{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{I}_{0}}}$ | $1.1$ | $[0,0,0]$ | $[1,0,0,0]$ | $[0,0,0]$
covariance | $1\mathrm{e}{-8}\times\mathbf{I}_{3}$ | $1\mathrm{e}{-4}\times\mathbf{I}_{3}$ | $1\mathrm{e}{-6}\times\mathbf{I}_{3}$ | $1\mathrm{e}{-4}$ | $5\mathrm{e}{-4}\times\mathbf{I}_{3}$ | $5\mathrm{e}{-5}\times\mathbf{I}_{3}$ | $5\mathrm{e}{-4}\times\mathbf{I}_{3}$
TABLE I: The table lists initial values and covariances setting for all pose
estimation experiments.

##  IV Inertial Dynamical Fusion Pipeline

###  IV-A Rotation Stage

We separate the rotation and translation estimation in a two-stage EKF
framework.

####  IV-A1 State

In the first rotation stage, there is only the rotation of the $\mathcal{I}$
in the $\mathcal{G}$ frame taken as filter’s state:

$\bm{x}={{}^{\mathcal{G}}_{\mathcal{I}}}\bm{q}.$ (12)

####  IV-A2 Process Model

The rotational equation is given:

$\displaystyle\bm{\dot{x}}=\frac{1}{2}{{}^{\mathcal{G}}_{\mathcal{I}}}\bm{q}\otimes{{}^{\mathcal{I}}}{\bm{\omega}}=\frac{1}{2}{{}^{\mathcal{G}}_{\mathcal{I}}}\bm{q}\otimes({{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}+{{\bm{n}}_{\bm{\omega}}}),$
(13)

where
${{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}={{}^{\mathcal{I}}}{\widetilde{\bm{\omega}}}-{{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{\omega}}$,
${{\bm{n}}_{\bm{\omega}}}\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{\omega}}^{2})$,
and ${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{\omega}}$ is the output of the
gyroscope De-Bias Net.

####  IV-A3 Measurement Model

The attitude controllers of most flying robots rely on the complementary
filters [25, 26] to obtain attitude observations. Similarly, we consider the
gravity alignment constraint to obtain the tilt observation:

$\displaystyle^{\mathcal{I}}\widehat{\bm{a}}\approx{{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}\bm{\mathrm{g}}+{{\bm{n}}_{\bm{a}}}.$
(14)

Similarly,
${{}^{\mathcal{I}}}{\widehat{\bm{a}}}={{}^{\mathcal{I}}}{\widetilde{\bm{a}}}-{{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{a}}$,
${{\bm{n}}_{\bm{a}}}\sim\mathcal{N}(0,\bm{\Sigma}_{\bm{a}}^{2})$, and
${{}^{\mathcal{I}}}\widehat{\bm{b}}_{\bm{a}}$ is the output of the accelerator
De-Bias Net.

###  IV-B Translation Stage

####  IV-B1 State

The state of the second stage is defined as:

$\bm{x}=({{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{B}}},~{}{{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}},~{}\tau,~{}\bm{d},~{}{{}^{\mathcal{I}}_{\mathcal{B}}}\bm{q},~{}{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}),$
(15)

where ${{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{B}}}$ and
${{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}$ are respectively
the velocity and position of the quadrotor body $\mathcal{B}$ frame expressed
in the $\mathcal{G}$ frame, $\bm{d}$ is the drag vector of
$\left(d_{x},d_{y},d_{z}\right)$, and
$({{}^{\mathcal{I}}_{\mathcal{B}}}\bm{q},{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}})$
is the extrinsic parameter between the $\mathcal{B}$ and the $\mathcal{I}$
frame.

####  IV-B2 Process Model

We regard the quadrotor dynamics as the input, and express the complete
process model as follows:

$\displaystyle{{}^{\mathcal{G}}}\bm{\dot{p}}{{}^{\mathcal{G}}_{\mathcal{B}}}$
$\displaystyle={{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}},$ (16)
$\displaystyle{{}^{\mathcal{G}}}\bm{\dot{v}}{{}^{\mathcal{G}}_{\mathcal{B}}}$
$\displaystyle=\frac{1}{m}{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}(\tau
U_{ss}\bm{e}_{3}-U_{s}D{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}+\widehat{\bm{f}}_{res}+{\bm{n}}_{\bm{f}})-{{}^{\mathcal{G}}}\bm{\mathrm{g}},$
$\displaystyle\dot{k}_{t}$
$\displaystyle=0,\quad\dot{\bm{d}}=\bm{0},\quad{{}^{\mathcal{I}}_{\mathcal{B}}}\bm{\dot{q}}=\bm{0},\quad{{}^{\mathcal{I}}}\bm{\dot{t}}{{}^{\mathcal{I}}_{\mathcal{B}}}=\bm{0},$

where
${{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}={{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}{{}_{\mathcal{B}}^{\mathcal{I}}}\mathbf{R}$,
and $\widehat{\bm{f}}_{res}$ and $\widehat{\bm{\Sigma}}_{\bm{f}}^{2}$
$({\bm{n}}_{\bm{f}}\sim\mathcal{N}(0,\widehat{\bm{\Sigma}}_{\bm{f}}^{2}))$ are
the Res-Dynamics Net outputs.

####  IV-B3 Measurement Model

Firstly, since accelerometer measurements are not used before, we take the
inconsistency of the ${\mathcal{I}}$ and ${\mathcal{B}}$ frame into account
and express the dynamics measurement equation as follow:

$\displaystyle{}^{\mathcal{I}}\widehat{\bm{a}}$
$\displaystyle=\frac{1}{m}{{}_{\mathcal{B}}^{\mathcal{I}}}\mathbf{R}\left(\tau
U_{ss}\bm{e}_{3}-U_{s}D{{}_{\mathcal{B}}^{\mathcal{G}}}\mathbf{R}^{\top}{{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}+\widehat{\bm{f}}_{res}\right)$
(17)
$\displaystyle-{{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}\times({{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}\times{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}})-{{}^{\mathcal{I}}}{\widehat{\bm{\alpha}}}\times{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}+{\bm{n}}_{a}.$

What’s more, our aforementioned V-P Net can be used as observers of velocity
and displacement:

$\displaystyle{{}^{\mathcal{G}}}\bm{\widehat{v}}{{}^{\mathcal{G}}_{\mathcal{I}}}$
$\displaystyle={{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}-{{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}\left({{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}\times{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}\right)+{\bm{n}}_{v},$
(18)
$\displaystyle{{}^{\mathcal{G}}}\bm{\widehat{p}}{{}^{\mathcal{G}}_{\mathcal{I}}}$
$\displaystyle={{}^{\mathcal{G}}}\bm{p}{{}^{\mathcal{G}}_{\mathcal{B}}}-{{}_{\mathcal{I}}^{\mathcal{G}}}\mathbf{R}{{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}+{\bm{n}}_{p}.$

Specially, V-P Net outputs
${{}^{\mathcal{G}}}\bm{\widehat{v}}{{}^{\mathcal{G}}_{\mathcal{I}}}$ and
${{}^{\mathcal{G}}}\bm{\widehat{p}}{{}^{\mathcal{G}}_{\mathcal{I}}}$ as well
as their covariances $\widehat{\bm{\Sigma}}_{\bm{v}}^{2}$
$({\bm{n}}_{\bm{v}}\sim\mathcal{N}(0,\widehat{\bm{\Sigma}}_{\bm{v}}^{2}))$ and
$\widehat{\bm{\Sigma}}_{\bm{p}}^{2}$
$({\bm{n}}_{\bm{p}}\sim\mathcal{N}(0,\widehat{\bm{\Sigma}}_{\bm{p}}^{2}))$.
More details about the EKF process and observability proof can be found in the
supplementary material [27].

##  V Experiments

###  V-A Metrics Definition

In order to assess the performance of our system, we define the metrics
including the Absolute Translation Error (ATE), the Absolute Rotation Error
(ARE), the Relative Translation Error (RTE), the Relative Rotation Error
(RRE), and the Translation Drift (TD), the Rotation Drift (RD) as follows:

  * •

ATE
(m)$:=\sqrt{\frac{1}{l}\sum_{i=1}^{l}\|{{}^{\mathcal{G}}}\bm{p}_{i}-{{}^{\mathcal{G}}}\bm{\widehat{p}}_{i}\|^{2}_{2}}$,

  * •

ARE
(m)$:=\sqrt{\frac{1}{l}\sum_{i=1}^{l}\|\bm{d}({{}^{\mathcal{G}}_{\mathcal{I}_{i}}}\bm{q},{{}^{\mathcal{G}}_{\mathcal{I}_{i}}}\bm{\widehat{q}})\|^{2}_{2}}$,

  * •

RTE
(%)$:=\sqrt{\frac{1}{l}\sum_{i=1}^{l}\|{{}^{\mathcal{G}}}\bm{p}_{i,i+\Delta
t}-{{}^{\mathcal{G}}}\bm{\widehat{p}}_{i,i+\Delta t}\|^{2}_{2}}$,

  * •

RRE (rad)$:=\sqrt{\frac{1}{l}\sum_{i=1}^{l}\|\bm{d}({{}^{\mathcal{I}_{i+\Delta
t}}_{\mathcal{I}_{i}}}\bm{q},{{}^{\mathcal{I}_{i+\Delta
t}}_{\mathcal{I}_{i}}}\bm{\widehat{q}})\|^{2}_{2}}$,

  * •

TD
(rad)$:=\|{{}^{\mathcal{G}}}\bm{p}_{l}-{{}^{\mathcal{G}}}\bm{\widehat{p}}_{l}\|^{2}_{2}/(\text{trajectory-
length})$,

  * •

RD
($\textrm{rad}/\min$)$:=\|\bm{d}({{}^{\mathcal{G}}_{\mathcal{I}_{l}}}\bm{q},{{}^{\mathcal{G}}_{\mathcal{I}_{l}}}\bm{\widehat{q}})\|^{2}_{2}/(\text{sequence-
duration})$,

where the rotation distance is $\bm{d}(\bm{q},\bm{\widehat{q}})$ =
$log(conj(\bm{\widehat{q}})\otimes\bm{q})$, $l$ is the length of each dataset
and $\Delta t$ is set to $1/20s$.

###  V-B Pose Comparison

To better demonstrate our system, we select several conventional and learning-
based algorithms to compare the estimation accuracy of 6D pose in the test
set. Note that we set the initial values and covariances in TABLE.I. The
proposed DIDO and TLIO [6] are the complete systems whose results are the
filter output. We train and test the former RoNIN-ResNet [2] by providing the
ground truth rotation, so there is no rotation error shown. There are two
conventional algorithms, Dyn-UKF [7] and Ori-EKF [28]. Dyn-UKF [7], integrates
the dynamics as a forward process and updates the tilt and velocity in
${\mathcal{B}}$ frame by the raw accelerometer measurements, while all states
are converted to the ${\mathcal{G}}$ frame by estimated rotation. Ori-EKF [28]
only obtains the rotation using the raw IMU measurements.

Fig. 4 shows the error distributions among the entire test set, which
distinctly shows the proposed DIDO outperforms other methods. Dyn-UKF [7]
based only on the momentum theory does not work well, while DIDO compensates
for the biased dynamics model resulting in a better performance. Fig. 5
presents three learning-based methods for position estimation on unseen simple
trajectories. Fig. 10 depicts the performance of different methods for each
axis. TLIO [6] and RoNIN-ResNet [2] both have unavoidable cumulative drift,
while attributing to the V-P Net, DIDO has stable position estimation.
Besides, De-Bias Net decelerates the severe yaw drift, which effectively helps
the position estimation.

Figure 4:  Pose comparison between DIDO and other methods. Each subplot shows
the cumulative density function (CDF) on the entire test set. The closer the
graph is to the left, the better the performance. Figure 5:  Top view diagrams
of on two test trajectories.

###  V-C Parameters Estimation

To show the effectiveness and robustness of the parameter identification
performance of the proposed system, we conduct an experiment by perturbing the
initial values of the estimated parameters. Fig. 6 presents the convergence of
the parameters in six independent runs, in which the majority of the
parameters can converge quickly, except for the extrinsic parameter rotation
expressed in terms of Euler angles ($XYZ$ in Tait–Bryan angles). This is due
to the fact that the quadrotor in our dataset does not fly fast enough, so the
yaw angle of the extrinsic rotation appears to be weakly observable. The other
degradation cases are: when ${{}^{\mathcal{I}}}{\widehat{\bm{\omega}}}$ =
$\bm{0}_{3\times 1}$,
${{}^{\mathcal{I}}}\bm{t}{{}^{\mathcal{I}}_{\mathcal{B}}}$ is unobservable,
and when ${{}^{\mathcal{G}}}\bm{v}{{}^{\mathcal{G}}_{\mathcal{B}}}$ =
$[0;0;*]$, $D$ and ${{}_{\mathcal{B}}^{\mathcal{I}}}\mathbf{R}$ are
unobservable. The detailed proof process is provided in the supplementary
material [27].

##  VI Ablation study

To be able to better illustrate the effectiveness of the proposed system, we
verify the behavior of each network module through ablation experiments.

###  VI-A De-Bias Net

We compare the rotation metrics for the gyroscope De-Bias Net and translation
metrics for the accelerometer one, respectively. As shown in the Fig. 7, the
debiased gyroscope significantly outperforms method without debiasing, and
updating by the debiased accelerometer results in better absolute rotation but
with a slight loss of relative rotation. The debiased accelerometer also
contributes to translation because debiasing facilitates the estimation of
velocity increment (the input of V-P Net) and thus improves the accuracy of
the V-P Net observation. Note that, the system is updated by Eq.(17) and
Eq.(18) when verifying the accelerometer De-Bias Net.

Figure 6: Parameters convergence for 6 runs. Figure 7: De-Bias Net ablation
study. The first row shows the rotational comparison between the three methods
net acc (debiased gyroscope ${\widehat{\bm{\omega}}}$ =
${\widetilde{\bm{\omega}}}$ \- $\widehat{\bm{b}}_{\bm{\omega}}$ integrating
and debiased accelerometer ${\widehat{\bm{a}}}$ = ${\widetilde{\bm{a}}}$ \-
$\widehat{\bm{b}}_{\bm{a}}$ updating), net gyr (${\widehat{\bm{\omega}}}$
integrating) and raw gyr (${\widetilde{\bm{\omega}}}$ integrating). The second
row shows the translational comparison between net acc (with accelerometer De-
Bias Net) and raw acc (without accelerometer De-Bias Net).

###  VI-B Res-Dynamics Net

The Res-Dynamics Net depends on dynamical parameters and other motion states
which need to be estimated online in the proposed system, so we validate the
tri-axial forces by comparing the complete EKF process with and without the
Res-Dynamics Net. Similarly, we also define a metric to measure the precision
of force estimation:

  * •

AFE$:=\sqrt{\frac{1}{l}\sum_{i=1}^{l}\|\bm{f}_{i}-\bm{\widehat{f}}_{i}\|^{2}_{2}}$.

As shown in the Fig. 8, Res-Dynamics Net allows for more accurate dynamics
modeling, especially in the z-axis.

###  VI-C V-P Net

To verify the effectiveness of the V-P Net, we design two sets of experiments.
As shown in the Fig. 9, P Net and V Net enhance the estimation accuracy of
position and velocity, respectively. Furthermore, using V-P Net results in
better RTE with guaranteed ATE in translation estimation.

###  VI-D One-stage or Two-stage

To demonstrate the effect of these three measurement models on the state
estimation system, we design four sets of experiments. As shown in the Fig.
11, rotation is prone to be incorrectly updated by each noisy observation in
the one-stage EKF, which further affects the estimation of velocity and
position. This is due to the fact that rotation is very sensitive as an
explicit or implicit input to the networks. Once the rotation can not be
updated correctly during the EKF process, it will make the outputs of networks
worse and thus impair the state estimation system.

Figure 8: Res-Dynamics Net ablation study. The metrics in three directions are
used to show the force estimation of net dyn and raw dyn (with and without
Res-Dynamics Net). Figure 9: V-P Net ablation study. Translational estimation
is compared under six different combinations of p (position update), v
(velocity update), a (acceleration update) and d (using the Res-Dynamics Net).

##  VII Discussions

###  VII-A Dynamics or IMU based

In this sensor fusion system, there are two ways to obtain acceleration, one
is the accelerometer measurements, and the other is the quadrotor dynamics.
However, if the accelerometer measurements are used as the process model, the
constraint between tilt and velocity does not hold. As a consequence, the
dynamics-based methods may be superior to the IMU-based one in the quadrotor
estimation system.

###  VII-B Covariance

Covariances are of great importance because they can be used to describe the
data distribution and also taken as weights for multi-sensor fusion. The
neural network covariances can dynamically depict the uncertainty of the
network output mean values, which reduce the inaccuracy led by the static
covariance setting. However, in the EKF process, the covariances may propagate
inaccurately due to the linear approximation of the process and measurement
models. Therefore, we do not directly use the neural network covariances but
scale the covariances in a similar way to TLIO to obtain a better estimation.

Figure 10: Demonstration of different methods for pose estimation. There are
two Random trajectories corresponding to different max uniaxial velocities
(a:0.5m/s, b:2.5m/s) in the test set. The ground truth positions are plotted
as black lines in the first row while ignoring the Dyn-UKF [7] due to its
divergence. The rotation errors are drawn in the second row. Figure 11: Pose
Comparison of EKF. One-stage and two-stage EKF are abbreviated as one and two,
respectively. p, v, a and d hold the same meanings as Fig. 9. For example, one
vd refers to a one-stage EKF updated by velocity in the case of the Res-
Dynamics Net.

##  VIII Conclusions and Future work

###  VIII-A Conclusions

In this work, we propose an inertial and quadrotor dynamical odometry system
introducing deep neural networks in a two-stage tightly-coupled EKF framework.
To the best of our knowledge, this is the first estimation framework that
combines IMU, quadrotor dynamics and deep learning methods to simultaneously
estimate kinematic, and dynamic states, as well as extrinsic parameters. To
take full advantage of the two interoceptive sensors, we design deep neural
networks to regress the IMU bias, the dynamics compensation, motion states of
the quadrotor, as well as their covariances.

Experimental results have demonstrated that the proposed method outperforms
the other conventional and learning-based methods in pose estimation. It is
accurate and invulnerable to consider De-Bias Net and gravity alignment
constraints for rotation estimation. And it is efficient and versatile to take
into account both quadrotor dynamics and deep neural networks for translation
estimation. The code and data will be open-sourced to the community
333https://zhangkunyi.github.io/DIDO/.

###  VIII-B Future work

Even though the proposed method makes use of deep neural networks to improve
the accuracy of estimation, it still does not guarantee a perfect globally
consistent yaw angle and position under long and highly maneuverable flight.
The plausible solution is to provide an absolute observation, such as GPS or
magnetic compass. Besides, some of the proposed modules can be supplied in
GPS-free environments with exteroceptive sensors to improve the robustness of
estimation. These additional sensors can be fused with the proposed method by
means of optimization or filtering.

## References

  * [1] C. Chen, X. Lu, A. Markham, and N. Trigoni, “IONet: Learning to cure the curse of drift in inertial odometry,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [2] S. Herath, H. Yan, and Y. Furukawa, “RoNIN: Robust neural inertial navigation in the wild: Benchmark, evaluations, and new methods,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 3146–3152.
  * [3] H. Yan, Q. Shan, and Y. Furukawa, “RIDI: Robust IMU double integration,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 621–636.
  * [4] M. Brossard, A. Barrau, and S. Bonnabel, “AI-IMU dead-reckoning,” _IEEE Transactions on Intelligent Vehicles_ , vol. 5, no. 4, pp. 585–595, 2020.
  * [5] S. Sun, D. Melamed, and K. Kitani, “IDOL: Inertial deep orientation-estimation and localization,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 7, 2021, pp. 6128–6137.
  * [6] W. Liu, D. Caruso, E. Ilg, J. Dong, A. I. Mourikis, K. Daniilidis, V. Kumar, and J. Engel, “TLIO: Tight learned inertial odometry,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5653–5660, 2020.
  * [7] J. Svacha, G. Loianno, and V. Kumar, “Inertial yaw-independent velocity and attitude estimation for high-speed quadrotor flight,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1109–1116, 2019.
  * [8] L. Bauersfeld, E. Kaufmann, P. Foehn, S. Sun, and D. Scaramuzza, “Neurobem: Hybrid aerodynamic quadrotor model,” _ROBOTICS: SCIENCE AND SYSTEM XVII_ , 2021.
  * [9] B. Rao, E. Kazemi, Y. Ding, D. M. Shila, F. M. Tucker, and L. Wang, “CTIN: Robust contextual transformer network for inertial navigation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 5, 2022, pp. 5413–5421.
  * [10] S. Roumeliotis and J. Burdick, “Stochastic cloning: a generalized framework for processing relative state measurements,” in _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292)_ , vol. 2, 2002, pp. 1788–1795.
  * [11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , vol. 30, 2017, pp. 5998–6008.
  * [12] R. Mahony, V. Kumar, and P. Corke, “Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor,” _IEEE Robotics and Automation magazine_ , vol. 19, no. 3, pp. 20–32, 2012.
  * [13] W. Khan and M. Nahon, “Toward an accurate physics-based UAV thruster model,” _IEEE/ASME Transactions on Mechatronics_ , vol. 18, no. 4, pp. 1269–1279, 2013.
  * [14] R. Gill and R. D’Andrea, “Propeller thrust and drag in forward flight,” in _2017 IEEE Conference on Control Technology and Applications (CCTA)_. IEEE, 2017, pp. 73–79.
  * [15] R. Gill and R. D’Andrea, “Computationally efficient force and moment models for propellers in UAV forward flight applications,” _Drones_ , vol. 3, no. 4, p. 77, 2019.
  * [16] B. Nisar, P. Foehn, D. Falanga, and D. Scaramuzza, “VIMO: Simultaneous visual inertial model-based odometry and force estimation,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 3, pp. 2785–2792, 2019.
  * [17] Z. Ding, T. Yang, K. Zhang, C. Xu, and F. Gao, “VID-Fusion: Robust visual-inertial-dynamics odometry for accurate external force estimation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 14 469–14 475.
  * [18] J. Svacha, J. Paulos, G. Loianno, and V. Kumar, “IMU-based inertia estimation for a quadrotor using newton-euler dynamics,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 3861–3867, 2020.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [20] D. Chen, N. Wang, R. Xu, W. Xie, H. Bao, and G. Zhang, “RNIN-VIO: Robust neural inertial navigation aided visual-inertial odometry in challenging scenes,” in _2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_. IEEE, 2021, pp. 275–283.
  * [21] A. Antonini, W. Guerra, V. Murali, T. Sayre-McCord, and S. Karaman, “The blackbird UAV dataset,” _The International Journal of Robotics Research_ , vol. 39, no. 10-11, pp. 1346–1364, 2020.
  * [22] K. Zhang, T. Yang, Z. Ding, S. Yang, T. Ma, M. Li, C. Xu, and F. Gao, “The visual-inertial- dynamical multirotor dataset,” in _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 7635–7641.
  * [23] D. Mellinger and V. Kumar, “Minimum snap trajectory generation and control for quadrotors,” in _2011 IEEE international conference on robotics and automation_. IEEE, 2011, pp. 2520–2525.
  * [24] P. Geneva and G. Huang, “vicon2gt: Derivations and analysis,” University of Delaware, Tech. Rep. RPNG-2020-VICON2GT, 2020. [Online]. Available: http://udel.edu/~ghuang/papers/tr_vicon2gt.pdf
  * [25] R. Mahony, T. Hamel, and J.-M. Pflimlin, “Nonlinear complementary filters on the special orthogonal group,” _IEEE Transactions on automatic control_ , vol. 53, no. 5, pp. 1203–1218, 2008.
  * [26] S. O. Madgwick, A. J. Harrison, and R. Vaidyanathan, “Estimation of IMU and MARG orientation using a gradient descent algorithm,” in _2011 IEEE international conference on rehabilitation robotics_. IEEE, 2011, pp. 1–7.
  * [27] K. Zhang, C. Jiang, J. Li, S. Yang, T. Ma, C. Xu, and F. Gao. (2022) Supplementary material: Deep inertial quadrotor dynamical odometry. [Online]. Available: https://github.com/zhangkunyi/DIDO/blob/main/DIDO_SupMat.pdf
  * [28] S. Sabatelli, M. Galgani, L. Fanucci, and A. Rocchi, “A double-stage kalman filter for orientation tracking with an integrated processor in 9-d IMU,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 62, no. 3, pp. 590–598, 2012.


# On Leakage of Code Generation Evaluation Datasets

Alexandre Matton, Tom Sherborne, Dennis Aumiller,
Elena Tommasone, Milad Alizadeh, Jingyi He,
Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon
Matthias Gallé
Cohere

###### Abstract

In this paper we consider contamination by code generation test sets, in
particular in their use in modern large language models. We discuss three
possible sources of such contamination and show findings supporting each of
them: (i) direct data leakage, (ii) indirect data leakage through the use of
synthetic data and (iii) overfitting to evaluation sets during model
selection.

Key to our findings is a new dataset of 161 prompts with their associated
python solutions, dataset which is released at
https://huggingface.co/datasets/CohereForAI/lbpp.

##  1 Introduction

Code generation has emerged as an important skill for large language models to
master. Measuring recent progress in code generation has relied on few,
critical benchmarks to judge performance between model families and
checkpoints. While many recent sophisticated evaluation datasets have been
proposed (Jain et al., 2024; Jimenez et al., 2024), the community largely
relies on HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) to
judge a new model’s code capability. In fact, all major announcements in
2023-2024 claiming advanced code capabilities—from academic and industry
labs—use at least one of these two datasets. Practically, reporting HumanEval
and MBPP is mandatory for a model to report competitive code generation.

However, the importance of these benchmarks has led to a conflict between
popularity and utility. On one side, obtaining competitive numbers comes with
significant scientific and economic reward—made increasingly easy with the
proliferation of public replicas of these datasets. However, this prevalence
has led to data leakage beyond the original evaluation scope, i.e., data
contamination, and once this evaluation data contaminates model training, the
validity of the metrics as a measure of generalization capability becomes
unreliable. If a model has been trained on the same data we use for out-of-
distribution generalization (or is selected based on its performance on that
data), we break an implicit tenet of how model capability can be measured. We
argue that understanding the effect of contamination is critical to accurately
interpreting scores on these benchmarks.

In this paper, we review the evidence that these two benchmarks have
contaminated most large LLMs, which we define as any procedure that leaked
those datasets during model training. The most obvious method of contamination
is presence inside training data, and we provide evidence that it is highly
probably that this occurs at a scale too large to be avoidable. A second
possibility is that contamination happens indirectly through the use of
synthetic data—a widespread paradigm used in particular to increase code
capabilities by generating additional code training tokens. Finally, we argue
that final model selection might have been overly influenced by their
performance on these datasets, overfitting to performance on these metrics
over general-purpose code-oriented skills.

To measure this contamination, we propose Less Basic Python Problems (LBPP), a
code generation benchmark similar to HumanEval and MBPP in style and scale,
but more difficult. LBPP is similarly portable, but is produced in a manner to
reduce any likelihood of leakage into present code training data. We
contribute LBPP to act as a genuinely held-out test set to measure current
code generation capability, and potential overfitting to HumanEval and MBPP.

##  2 Related Work

HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) remain the most
reported results on public leaderboards, but others similar datasets exist
(Hendrycks et al., 2021; Li et al., 2022). They consist of short and mostly
simple (not programming competition level) instructions with completions in
Python. Translation into other programming languages exist for those datasets
(Muennighoff et al., 2023; Cassano et al., 2022), as well as versions with
additional tests (Liu et al., 2024).

(Jain et al., 2024) proposed a continuously updated set of leetcode to improve
dataset challenge by including harder and novel (unseen) prompts. (Jimenez et
al., 2024) aims for challenging software engineering problems, that require
understanding of full repositories. In a similar vein,
RepoQA111https://github.com/evalplus/repoqa and Bug In The Code
Stack222https://github.com/HammingHQ/bug-in-the-code-stack focus on
understanding long contexts within code tasks. One proposed solution is to use
hidden evaluation sets (Zhang et al., 2024), however, these do not allow
inspection of failure cases and requires trusting the quality and correctness
of an opaque ‘black-box’ evaluation setup. Recently, Riddell et al. (2024)
analyzed data contamination in popular pretraining datasets: reporting that
$12.2\%$ of HumanEval samples are present in The Pile (Gao et al., 2020), and
$18.9\%$ in The Stack (Kocetkov et al., 2022). Different from our analysis
however, they conclude: “we do not find the performance rankings of the models
to change with decontaminated results”.

(a) HumanEval

(b) MBPP

Figure 1: Pass@1 rate of popular datasets and the 161 prompts in LBPP.

##  3 Possible sources of contamination

We provide three hypotheses—with evidence for each one—on why existing models
might be over-optimized towards existing leaked benchmarks.

###  3.1 Direct data leakage

The most obvious reason is the simplest: many of the test datasets are of
widespread use and the simplest answer might be that modern LLMs are just
trained on this evaluation data. We note that intentional (i.e., to cheat) or
unintentional contamination has the same net effect: training on evaluation
data limits the confidence and utility of the benchmark results. For code
tasks, it is very expensive to curate datasets of natural language to code
instructions (one example generally costing several dozen US dollars). For any
group aiming to minimize data cost to improve coding performance, the dollar
value of creating new datasets can be very high. This leads to a common
practice of web scraping code-oriented resources (e.g., GitHub or
Stackoverflow) for data. However, these resources are also likely sources of
contamination. Since the release of HumanEval and MBPP in 2021 these datasets
have been branched, re-used and copied all across the Internet. The small data
size and portability of such benchmarks encourages replication within code
repositories. For example, searching for the prompts from HumanEval on GitHub
returns a hit in all cases—the median hits is $99$ and the minimum $43$ (see
Fig. 2). In many cases, these hits are exact duplicates and indications of a
fork of the original dataset.

While decontamination of training sets is becoming more common, present
decontamination filters designed for natural text adapts poorly to code. To
operate efficiently at scale, most filters rely on generic deduplication
algorithms e.g., such as $n$-gram matching or hashing functions Lee et al.
(2022). Such surface-level matching does not adequately capture code
similarity where a simple variable name change leaves program semantics
unchanged, but changing a single keyword can have profound changes.333E.g.,
compare the instruction “return true if the string is a float” with “return
true if the string is a verb”. The same shortcomings of decontamination
efforts apply to the creation of large-scale synthetic datasets: for example
the model-generated dataset of Starcoder (Li et al., 2023) is decontaminated
only by removing exact docstrings or solutions that match HumanEval or MBPP.

Figure 2: Histogram (excluding outliers) of occurrences for HumanEval prompts
in public GitHub repositories. Every prompt occurs at least $43$ times.

The recent exploration of Riddell et al. (2024) aims to quantify the
proportion of this data leakage in existing datasets using plagiarism tools
specifically designed for code. Even when static training datasets are
cleaned, contamination may persist. Entities who serve models through an API
may encounter these benchmark tasks when evaluated by third party users. When
a sample of real model usage is annotated for future training data, samples
from benchmark evaluation can leak into future training corpora. Furthermore,
these samples may include subtle phrasing variations and format changes that
further complicate heuristic deduplication. In this scenario, a model may
easily memorize completions to purportedly novel prompts. As evidence of this
phenomena, we prompted one popular commercial system with partial prompts from
HumanEval that were designed to keep the instruction under-specified. Table
Appendix in the Appendix shows the outcome and evidence that—despite the
ambiguity of the prompt—the resulting completion matches exactly the gold
solution from the test set of HumanEval.

###  3.2 Data leakage through synthetic data

The most capable of code language models rely heavily on the use of synthetic
training data (Xu et al., 2023; Wei et al., 2023, 2024).

A typical pipeline generally consists of: curating prompts related to code
generation, inferring completions with a previously trained LLM, and
synthesizing unit tests for relevant prompts using LLMs. Completions that pass
the respective unit tests are considered valid code solutions and can be used
as future training examples. Alternatively, if a sufficiently powerful model
is used, completions might be used as-is.

evol-instruct for example comprises $110k$ complex query prompts coupled with
completions from numerous closed and open-source models.444Per downloads, the
most popular version is a ‘lightly decontaminated’ version on HuggingFace
here. It is widely used by many code LLMs such as WizardCoder (Xu et al.,
2023). Prior reports (Yu et al., 2023, page 8), (Wei et al., 2023, page 4)
discuss an apparent high similarity between some of those examples in evol-
instruct and HumanEval. We extend this analysis by studying the similarity
between embedded representations of the prompts555Embedded using Cohere embed
v3 (Team, 2024). of HumanEval and MBPP with nearest neighbors from evol-
instruct and Starcoder-Instruct. Fig 3 highlights widespread similarity
between evaluation data and synthetic training datasets ‘Starcoder V2 OSS
Instruct’ and ‘evol instruct’. Even if unintentional, this contamination
further damages the utility of these benchmarks as a held-out test of code
generation capability.

Figure 3: Histogram of cosine similarities for prompts in HumanEval, MBPP and
LBPP relative to two popular synthetic code training datasets. We note the
high similarity between most HumanEval prompts to evol-instruct, and how LBPP
has reduced overall similarity to either training dataset.

Strikingly, we saw that training on this dataset can increase performance of
one of our models by 14 absolute points on HumanEval (from 0.52 pass@1 to
0.66), while only by 1 point on MBPP (from 0.52 to 0.53). Inspecting closest
neighbours for almost all examples in HumanEval (see examples in Table
Appendix in the Appendix)—we identify a semantically equivalent version in
evol-instruct.

Even when the original evaluation datasets were not used as inspiration for
the creation of the synthetic datasets the simple nature of the evaluation
datasets might make duplication unavoidable. There are only so many short
natural language prompts describing typical interview-style programming
questions that can be used. While synthetically generated prompts may not be
explicitly based on provided examples from a given test set in the prompt
context, the massive scale of these datasets ($238k$ instances for StarCoder-
Instruct prior to deduplication) runs the risk of exhausting the possible
number of variations on questions. Table 1 shows examples where generated
prompts are extremely similar to questions in the MBPP test set. Note that
despite that similarity training on this dataset did not improve substantially
performance on MBPP.

###  3.3 Overfitting to test sets

The exaggerated importance of these benchmarks encourages an incentive
structure where model selection prioritizes gain on a narrow suite of metrics.
While it may be tempting to use such benchmarks as a deciding factor between
similar checkpoints, there is weak evidence for these benchmarks correlating
with ‘solving code generation’. While the meaning and measurement of this
unscientific objective is subject to constant revision, selecting for optimal
HumanEval performance may be akin to $p$-hacking in other fields. This
practice can be justified by assuming that these benchmarks are the new dev
sets, while the true test is the usage of users over time. The risk remains
however that some models overfit to those test sets more than others,
distorting perception on relative performance.

In order to measure this, we created Less Basic Python Problems (LBPP), a
dataset of 161 code completion problems in the style of HumanEval. Human
annotators were instructed to create totally fresh problems, which were not
solvable by an internal model666We will update references to internal models
in future revisions. they had access to. Annotators had competitive
programming experience and could use programming books as inspiration, but
were instructed not to copy existing solutions on the Internet and not to use
any LLMs. All annotators were paid above minimum wage in their respective
countries, and all final prompt-completion pairs were manually reviewed by the
authors. This adversarial collection resulted in more difficult problems, with
most models solving less than $50\%$ of the dataset. Results on a selection of
models are in Table 4 in the Appendix.

Using this dataset we can correlate the performances of existing models on
them against performance of the well-known benchmarks. The two plots comparing
them to HumanEval and MBPP can be seen in Fig. 1. There is a clear correlation
on both data-sets, indicating that the public benchmarks are still a valuable
target signal. However, when zooming in — in particular for models from the
same family — the correlation often becomes negative, which might indicate
that the selected checkpoint to release performs better on those public
datasets while under-performing on new datapoints. Note in particular in Fig.
1(a) the crowded space between 0.75 and 0.8 of the $x$-axis (pass@1 of
HumanEval). Despite very similar performance on that public evaluation set,
the numbers of LBPP vary wildly. On the other side, in Fig. 1(b) a vertical
line can be traced at around $0.38$ of LBPP that crosses various points –
indicating maybe an overfit to MBPP.

##  4 Conclusion

We study the cause and effect of data contamination by two popular code
generation benchmarks. Our analysis highlights that contamination is likely
unavoidable at the LLM scale given the difficulty of filtering every potential
permutation of a benchmark dataset. This insight motivates our contribution of
a novel code generation benchmark to evaluate contemporary LLMs in a
contamination-free setting. We are well aware that our decision to release
this dataset will make future leakage impossible to control. However, with the
context of the fast-paced model development cycles that LLMs are currently
undergoing we believe that releasing this increases trustworthiness and
usefulness of this dataset. It is conveniently designed to serve as drop-in
replacement (or addition) of current evaluation sets. On top of its newness,
the more challenging nature of this dataset also allows it to provide more
signal for model comparison.

##  5 Limitations

All the model analysis was done black-box, without inspecting the model
weights or the training set (excepting the work on synthetic data). There is
no reason why this dataset will not follow the same path than the two studied
here. As mentioned in the Conclusion we believe there is more value in that
than in an alternative solution (not releasing or keeping it behind an API
access).

## References

  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021.  Program synthesis with large language models.  _CoRR_ , abs/2108.07732.
  * Cassano et al. (2022) Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. 2022.  Multipl-e: A scalable and extensible approach to benchmarking neural code generation.  _arXiv preprint arXiv:2208.08227_.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021.  Measuring coding challenge competence with apps.  _arXiv preprint arXiv:2105.09938_.
  * Jain et al. (2024) Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024.  Livecodebench: Holistic and contamination free evaluation of large language models for code.  _arXiv preprint arXiv:2403.07974_.
  * Jimenez et al. (2024) Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024.  SWE-bench: Can language models resolve real-world github issues? In _The Twelfth International Conference on Learning Representations_.
  * Kocetkov et al. (2022) Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022.  The stack: 3 tb of permissively licensed source code.  _arXiv preprint arXiv:2211.15533_.
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022.  Deduplicating training data makes language models better.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8424–8445, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097.
  * Liu et al. (2024) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.  _Advances in Neural Information Processing Systems_ , 36.
  * Muennighoff et al. (2023) Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023.  Octopack: Instruction tuning code large language models.  _arXiv preprint arXiv:2308.07124_.
  * Riddell et al. (2024) Martin Riddell, Ansong Ni, and Arman Cohan. 2024.  Quantifying contamination in evaluating code generation capabilities of language models.  _arXiv preprint arXiv:2403.04811_.
  * Team (2024) Cohere Embedding Team. 2024.  Cohere embed-english-v3.0.
  * Wei et al. (2024) Yuxiang Wei, Federico Cassano, Yifeng Ding, Naman Jain, Harm de Vries, Leandro von Werra, Arjun Guha, and Lingming Zhang. 2024.  Starcoder2-instruct: Fully transparent and permissive self-alignment for code generation.
  * Wei et al. (2023) Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023.  Magicoder: Source code is all you need.  _arXiv preprint arXiv:2312.02120_.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2023.  Wizardlm: Empowering large pre-trained language models to follow complex instructions.  In _The Twelfth International Conference on Learning Representations_.
  * Yu et al. (2023) Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. 2023.  Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation.  _arXiv preprint arXiv:2312.14187_.
  * Zhang et al. (2024) Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. 2024.  A careful examination of large language model performance on grade school arithmetic.  _arXiv preprint arXiv:2405.00332_.

## Appendix

MBPP test prompt |  evol-instruct prompt
---|---
Write a function to find the perimeter of a square |  Calculate the perimeter of a square
Write a python function to find the volume of a triangular prism |  How can I use JavaScript to calculate the volume of a triangular prism with an irregular base and uneven side lengths?
Write a function to convert snake case string to camel case string. |  Convert the sentence to camel case.
Table 1: Examples of similar prompts from the test set of MBPP found in
evol_instruct (Xu et al., 2023). Table 2: Original human evaluation prompts
with the completion from a major LLM provider.

Original human-eval prompt |  Submitted Prompt |  Opus completion
---|---|---
⬇ def is_happy(s): """You are given a string s. Your task is to check if the string is happy or not. A string is happy if its length is at least 3 and every 3 consecutive letters are distinct For example: is_happy(a) => False is_happy(aa) => False is_happy(abcd) => True is_happy(aabb) => False is_happy(adb) => True is_happy(xyy) => False """ |  Continue this code. Do not add any test, just continue it:
⬇ def is_happy(s): """ You are given a string s. Your task is to check if the str |  Here’s the continuation of the code without adding any tests: ⬇ def is_happy(s): """ You are given a string s. Your task is to check if the string is happy. A string is called happy if its length is at least 3 and every 3 consecutive letters are distinct [...]
⬇ def count_upper(s): """ Given a string s, count the number of uppercase vowels in even indices. For example: count_upper(’aBCdEf’) returns 1 count_upper(’abcdefg’) returns 0 count_upper(’dBBE’) returns 0 """ |  Complete this instruction, that is taken from the Human Eval test set: ⬇ def count_upper(s): """ Given a string s, count the number of |  Here is the completion of the given function signature and docstring to count the number of uppercase letters in a string: ⬇ def count_upper(s): """ Given a string s, count the number of uppercase vowels in the string. [...]
⬇ def any_int(x, y, z): ’’’ Create a function that takes 3 numbers. Returns true if one of the numbers is equal to the sum of the other two, and all numbers are integers. Returns false in any other cases. Examples: any_int(5, 2, 7) -> True any_int(3, 2, 2) -> False any_int(3, -2, 1) -> True any_int(3.6, -2.2, 2) -> False ’’’ |  continue this code. ⬇ def any_int(x, y, z): """Create a function that takes 3 numbers. Return true i |  ⬇ def any_int(x, y, z): """ Create a function that takes 3 numbers. Returns true if one of the numbers is equal to the sum of the other two numbers. Otherwise returns false. """ [...]
⬇ def order_by_points(nums): """ Write a function which sorts the given list of integers in ascending order according to the sum of their digits. Note: if there are several items with similar sum of their digits, order them based on their index in original list. For example: order_by_points([1, 11, -1, -11, -12]) == [-1, -11, 1, -12, 11] order_by_points([]) == [] """ |  You will be given an instruction to complete a python function. This instruction is cut in half, so you need to complete it. No need of generating the final code, only the instruction is required. The incomplete instruction is: ⬇ def order_by_points(nums): """ Write a function which sorts the given list of integers in ascending order according |  Here’s the completed instruction for the Python function: ⬇ def order_by_points(nums): """ Write a function which sorts the given list of integers in ascending order according to the sum of their digits. If two integers have the same sum of their digits, order them based on their index in the original list. """

Table 3: Most similar prompt in evol-instruct for a non-handpicked selection
of HumanEval prompts.

HumanEval prompt |  Closest evol-instrct prompt
---|---
⬇ Write a Python function ‘longest(strings: List[str]) -> Optional[str]‘ to solve the following problem: Out of list of strings, return the longest one. Return the first one in case of multiple strings of the same length. Return None in case the input list is empty. >>> longest([]) >>> longest([’a’, ’b’, ’c’]) ’a’ >>> longest([’a’, ’bb’, ’ccc’]) ’ccc’ |  ⬇ Complete the code below, considering an augmented depth of inquiry and maintaining brevity: from typing import List, Optional def longest(strings: List[str]) -> Optional[str]: """ From a list of strings, return the longest one. For multiple strings with equal length, return the first. For an empty list, return None. >>> longest([]) >>> longest([’a’, ’b’, ’c’]) ’a’ >>> longest([’a’, ’bb’, ’ccc’]) ’ccc’
⬇ Write a Python function ‘make_a_pile(n)‘ to solve the following problem: Given a positive integer n, you have to make a pile of n levels of stones. The first level has n stones. The number of stones in the next level is: - the next odd number if n is odd. - the next even number if n is even. Return the number of stones in each level in a list, where element at index i represents the number of stones in the level (i+1). Examples: >>> make_a_pile(3) [3, 5, 7] |  ⬇ Please complete the following code with added difficulty: def make_a_pile(n, pattern): """ Given a positive integer n, you have to make a pile of n levels of stones. The first level has n stones. The number of stones in the next level is determined by the given pattern ’odd’ or ’even’: - If pattern is ’odd’, add the next odd number to the previous level stones. - If pattern is ’even’, add the next even number to the previous level stones. Return the number of stones in each level in a list, where element at index i represents the number of stones in the level (i+1). Examples: >>> make_a_pile(3, ’odd’) [3, 5, 7] >>> make_a_pile(3, ’even’) [3, 6, 9] """
⬇ Write a Python function ‘x_or_y(n, x, y)‘ to solve the following problem: A simple program which should return the value of x if n is a prime number and should return the value of y otherwise. Examples: for x_or_y(7, 34, 12) == 34 for x_or_y(15, 8, 5) == 5 |  ⬇ Complete the subsequent lines of programming code: /* An elementary program intended to, given a single n value, decipher between two distinct possibilities: - If the provided n is a numerical prime, the program should yield the output value equivalent to variable x. - Should the n fail to meet the prime criteria, the program should then present the value contained within variable y as its output. Examples: Upon running x_or_y(7, 34, 12) the output will be 34 While executing x_or_y(15, 8, 5) the output will be 5 */ #include<stdio.h> using namespace std; int x_or_y(int n,int x,int y){

Model Name | Family | pass@1
---|---|---
Mistral-7b | Mistral | 0.11
Mixtral 8x7B | Mistral | 0.17
Mixtral 8x22B | Mistral | 0.38
Codestral 22B | Mistral | 0.40
Codellama-7b-instruct | Meta | 0.14
Codellama-34b-instruct | Meta | 0.19
Llama2-7b-chat | Meta | 0.02
Llama2-70b-chat | Meta | 0.10
Llama3-8b-instruct | Meta | 0.27
Llama3-70b-instruct | Meta | 0.49
GPT-3.5-tb | OpenAI | 0.37
GPT-4 | OpenAI | 0.53
GPT-4o | OpenAI | 0.63
Claude-2 | Anthropic | 0.34
Claude-3-haiku | Anthropic | 0.34
Claude-3-sonnet | Anthropic | 0.40
Claude-3-opus | Anthropic | 0.54
Deepseek-coder-33b-instruct | DeepSeek | 0.39
DBRX Instruct | DBRX | 0.25
Qwen1.5-72B-Chat | Qwen | 0.20
Qwen1.5-110B-Chat | Qwen | 0.30
Qwen 2-72B Instruct | Qwen | 0.42
Command R+ | Cohere | 0.22
Command R | Cohere | 0.12
Table 4: Results of pass@1 rate on the LBPP dataset for a selection of models.