# An Empirical Bayes Analysis of Vehicle Trajectory Models

Yue Yao, Daniel Goehring, Joerg Reichardt Yue Yao is with Continental AG and
Ph.D. candidate at Freie Universität Berlin.Daniel Goering is Juniorprofessor
for Autonomous Vehicles at the Dahlem Center for Machine Learning and Robotics
of the Freie Universität BerlinJoerg Reichardt is with Continental AG.

###### Abstract

We present an in-depth empirical analysis of the trade-off between model
complexity and representation error in modelling vehicle trajectories.
Analyzing several large public datasets, we show that simple linear models do
represent real-world trajectories with high fidelity over relevant time scales
at very moderate model complexity. This finding allows the formulation of
trajectory tracking and prediction as a Bayesian filtering problem. Using an
Empirical Bayes approach, we estimate prior distributions over model
parameters from the data that inform the motion models necessary in the
trajectory tracking problem and that can help regularize prediction models. We
argue for the use of linear models in trajectory prediction tasks as their
representation error is much smaller than the typical epistemic uncertainty in
this task.

##  I INTRODUCTION

Safe and comfortable driving necessitates anticipatory planning. Drivers must
form expectations about the future development of the traffic scene and adapt
their behavior accordingly. Other traffic participants’ (TPs) motion plans are
unobservable, and their intents are often not signalled unequivocally, even
when required by law. The expectation on the future trajectory of TPs is thus
uncertain and must be modelled probabilistically. Past trajectories are
equally uncertain, albeit to a lesser degree, as observations are generally
noisy or may be missing entirely due to occlusions.

In order to formulate probabilistic statements about traffic scenarios, both
past and future trajectories must be considered and modelled as random
variables. This naturally leads to the question of how to best represent a
trajectory for prediction and tracking applications. In this contribution, we
present an empirical investigation into the usefulness of one representation
that is mathematically particularly appealing: linear combinations of basis
functions.

Our discussion will proceed as follows: We will first summarize some of the
theoretical advantages and disadvantages of sequence-based trajectory
representations and compare them with classical linear combinations of basis
functions. We proceed to investigate the expressive power of this trajectory
representation by analyzing a number of publicly available datasets of vehicle
trajectories. Specifically, we employ the Empirical Bayes method to estimate
prior distributions over model parameters and observation noise. We
characterize the trade-off between model complexity, i.e. the number of basis
functions used, and data-fit. For this, we employ information theoretic
measures and mean absolute spatial error between the trajectory representation
and the data. We investigate this trade-off for trajectories of various
lengths and determine the optimal model complexities before concluding with a
short summary of our findings.

##  II Related Work

###  II-A Sequence Models of Trajectories

Trajectory prediction systems must take representations of past object
trajectories and the current static environment in a scene as input, and
output some representation of a predicted future trajectory over a finite time
horizon $T$.

Past information can be input as a sequence of observations at fixed temporal
or spatial resolution. One popular approach for items in such an input
sequence is rasterization, which renders past object trajectories and the
static environment into a stack of images from a “bird’s-eye” view [1, 2].
Rasterizing the environment is powerful and comprehensive in representing
arbitrary information as a multi-channel image. However, the memory usage and
computational cost need to scale with desired spatial and temporal resolution
and the discrete nature of this approach is inadequate in representing
continuous physical states.

Alternatively, the input sequence contains object trajectories as lists of
kinematic measurements (position, heading, velocity, etc.). The contextual
static environment can also be represented in form of lists of static features
such as road boundary positions [3] or in form of a single ’bird’s-eye’
rendering of the scene [4]. The sequence-wise representation is a more compact
form compared to rasterization. However, this approach still represents
discrete physical states and the memory usage scales with the resolution of
this approach.

In symmetry with the sequential nature of the input, many prediction systems
employ sequence representation also for the output, e.g., as a sequence of
positions [5, 4, 6] or control states (acceleration and turn rate) [7, 8] at
fixed time-points over the prediction horizon. The computational requirements
of these approaches scale with the length and temporal resolution of the
prediction horizon.

Sequence-based approaches are examples of unbiased models that offer great
flexibility of expression. They can express even random, erratic trajectories.
The downside of unbiasedness of any estimator is the high variance which in
turn generally requires larger amounts of training data and the fact that
sequence-based representations amalgamate the observation uncertainty and the
underlying system behavior to be modelled.

###  II-B Linear Models of Trajectories

On the other hand, linear combinations of basis functions limit their
expressive power to what the basis functions allow - they are the foundation
of compact but biased estimators. They may introduce systematic error (bias)
but feature less variance and typically reduce the need for training data. We
will turn to a detailed discussion of their properties and merits next.

In contrast to paths, i.e. curves in space, trajectories are curves in space
_and_ time. Over any finite time horizon $[t_{0},t_{0}+T]$, the position of an
object $\mathbf{c}(\tau)\in\mathbb{R}^{d}$ at (rescaled) time
$\tau=(t-t_{0})/T$ can be expressed as a linear combination of $n+1$ _fixed_
basis functions of time $\phi_{k}(\tau):\mathbb{R}\rightarrow\mathbb{R}$ and
variable points in space $\boldsymbol{\omega}_{k}\in\mathbb{R}^{d}$
parameterizing the curve:

$\mathbf{c}(\tau)=\sum_{k=0}^{n}\phi_{k}(\tau)\boldsymbol{w}_{k}.$ (1)

It allows us to incorporate prior knowledge about object trajectories via the
basis functions $\phi_{k}(\tau)$. For example, since traffic participants
carry mass and underlie physical constraints on how much force they can exert
as well as strong preferences for smooth motion [9, 10, 11], we can choose a
small number $n+1$ of smooth basis functions and model deviations from smooth
behavior as random noise. Authors of [12, 13] all use polynomials of at most
degree 4 as basis functions.

In contrast to sequence-based trajectory representations, this formulation is
temporally continuous, i.e. provides arbitrary temporal resolution as
highlighted by Su et al. [13].

Parameters $\boldsymbol{\omega}_{k}$ have spatial semantics, i.e. they can be
interpreted as points in space and this allows to formulate prior
distributions based on spatial information, which is advantageous to
regularize and inform predictions of TPs future motion [14]. Without loss of
generality, we can constrain the range of $\phi_{k}(\tau)$ to the interval
$[0,1]$ and this automatically constrains $\mathbf{c}(\tau)$ to the convex
hull of the $\boldsymbol{\omega}_{k}$.

Parameters $\boldsymbol{\omega}_{k}$ transform lossless as spatial points
under translation and rotation of the coordinate reference frame, which
significantly simplifies applications with moving sensors.

Temporal derivatives only affect the basis functions
$\dot{\mathbf{c}}(\tau)=\sum_{k=0}^{n}\dot{\phi}_{k}(\tau)\boldsymbol{\omega}_{k}$,
i.e. positions, velocities, accelerations and higher derivatives share the
same parameterization!

Spatio-temporal distributions over the kinematic state of an object at any
point in time can be readily derived from distributions over the parameters
$\boldsymbol{\omega}_{k}$.

We may estimate the parameters $\boldsymbol{\omega}_{k}$ via the solution of a
linear system from either measurements of position at different time-points or
measurements of different derivatives at the same time-point, or combinations
thereof. For example, with $n=5$ and $\phi_{k}(\tau)=\tau^{k}$, we only need
position, velocity and acceleration at two time-points $\tau$ and
$\tau+\Delta\tau$ in order to fully determine all $6$ trajectory parameters
$\boldsymbol{\omega}_{k}$.

The linear combination of basis functions enables the drop-in replacement of
kinematic object states by trajectories over constant time horizons in
tracking and filtering applications at minimal computational overhead. It even
allows the formulation of the trajectory prediction problem as a filtering
problem [15]. In these applications, the trajectory parameters
$\boldsymbol{\omega}_{k}$ replace the conventionally used kinematic state
variables for tracking, filtering and prediction. The motion models in such
applications can then be informed and regularized by prior distributions over
$\boldsymbol{\omega}_{k}$.

However, in order to reap the benefits of these mathematical merits in
practical applications, we must show that linear combinations of a few smooth
basis functions can indeed represent real-world trajectories with high
fidelity.

Our key contributions are thus as follows: we provide a methodology to i)
estimate the optimal model complexity $\hat{n}$ from noisy data, ii) estimate
prior distributions over observation noise and model parameters
$\boldsymbol{\omega}_{k}$ to regularize model fits and trajectory predictions,
and iii) provide an extensive empirical investigation of the representation
error of this trajectory formulation for the case of polynomial basis
functions.

Note that this choice of basis functions is quite general but not unique and
only illustrates the general method of estimating prior distributions and
model complexity111The astute reader will recognize the choice of Bernstein
Polynomials for $\phi_{k}(\tau)$ leads to the familiar Bezier curves. In fact,
all formulations with polynomial basis functions of degrees $0,..,n$ are
equivalent and can be interchanged via a fixed linear transformation of the
model parameters $\boldsymbol{\omega}_{k}$.

##  III Estimation of Representation Error

(a)

(b)

Figure 1: a) A typical scene from a trajectory prediction dataset, here
Argoverse Motion Forecasting v1.1 [16]. Data is gathered by a moving sensor
platform (ego vehicle) and subsequently transformed into a fixed world
coordinate frame. As distance and angle between sensor and agent change during
recording, the observation covariance of agent locations stretches and rotates
over time. We show all sample points and a few $95\%$ confidence ellipses for
agent position, enlarged by a factor of $4$ for better visibility. b) The same
agent trajectory as in a) but fitted with a polynomial trajectory
representation estimated via eq. (2). The resulting posterior covariances for
agent positions are also shown, enlarged by a factor of $8$ for better
visibility.

Datasets for training and evaluating trajectory prediction methods such as
Argoverse Motion Forecasting v1.1 (A1) [16], Argoverse 2 Motion Forecasting
(A2) [17] and Waymo Open Motion (WO) [18] are obtained from a moving sensor
platform (_ego_ vehicle) during measurement campaigns. Object detections are
tracked and transformed into a _world_ coordinate system and reported in the
dataset. Each dataset selects one or multiple objects of interest in one
scenario and refers to them as _agents_. Rather than ground truth, object
positions and kinematics in the data represent noisy estimates. Hence, when
fitting models to the data in order to measure the approximation error, we
need to regularize and take the observation noise into account explicitly,
i.e. we need to perform a Bayesian regression. The kinematic variables
provided in datasets vary, but all provide position measurements for object
center points, which is what we focus on here. Figure 1a shows a typical
example.

TABLE I: Datasets characteristics Dataset (training split) |  A1 |  A2 |  WO
---|---|---|---
#scenarios, #ego trajectories |  206K |  200K |  487K
#agent (vehicle) trajectories |  206K |  176K |  1.84M
maximal time horizon [s] |  5 |  11 |  9
#cities |  2 |  6 |  6
sampling rate | 10 Hz
trajectory information | position |  2D |  2D |  3D
velocity |  - |  2D |  2D
orientation |  - |  ✓ |  ✓
timestamp |  ✓ |  ✓ |  ✓
|  |  |  |

In order to formulate the regression, we will introduce some definitions for
notational convenience. We form a parameter vector
$\boldsymbol{\omega}\in\mathbb{R}^{(n+1)d}$ as
$\boldsymbol{\omega}=[\boldsymbol{\omega}_{0},\cdots,\boldsymbol{\omega}_{n}]^{\mathrm{T}}$.
Then we can express a Gaussian prior over parameters
$p(\boldsymbol{\omega})=\mathcal{N}(\boldsymbol{\omega}|\boldmath{0},\boldsymbol{\Sigma}_{\boldsymbol{\omega}})$.
The prior has zero mean due to symmetry and a full covariance
$\boldsymbol{\Sigma}_{\boldsymbol{\omega}}$ that allows to express
correlations between spatial dimensions. We form a vector of basis functions
$\boldsymbol{\phi}(\tau)\in\mathbb{R}^{n+1}$ by concatenation
$\boldsymbol{\phi}(\tau)=[\phi_{0}(\tau),\cdots,\phi_{n}(\tau)]^{\mathrm{T}}$.
In this notation, equation (1) is then written as
$\mathbf{c}(\tau)=(\boldsymbol{\phi}^{\mathrm{T}}(\tau)\otimes\mathbf{I}_{d})\boldsymbol{\omega}$,
where $\mathbf{I}_{d}$ is a $d\times d$ identity matrix and the Kronecker
product $\otimes$ distributes the basis functions over the spatial dimensions.
Data for a single object $i$ consists of $m$ sample points at discrete time-
points $\tau_{i,1},...,\tau_{i,m}$. We can arrange all sample points of object
$i$ into a vector $\mathbf{c}_{i}\in\mathbb{R}^{md}$ as
$\mathbf{c}_{i}=[\mathbf{c}_{i}(\tau_{i,1}),\cdots,\mathbf{c}_{i}(\tau_{i,m})]^{\mathrm{T}}$
noting that the sample times in general are different for different objects.
By introducing the $(n+1)d\times md$ matrix
$\boldsymbol{\Phi}_{i}=[\boldsymbol{\phi}(\tau_{i,1})\otimes\mathbf{I}_{d},\cdots,\boldsymbol{\phi}(\tau_{i,m})\otimes\mathbf{I}_{d}]$
we can write most compactly
$\mathbf{c}_{i}=\boldsymbol{\Phi}_{i}^{\mathrm{T}}\boldsymbol{\omega}$. We
denote the measurement of trajectory with $\mathbf{c}^{\mathrm{ob}}_{i}$ for
$i^{\mathrm{th}}$ object and $\mathbf{c}^{\mathrm{ob}}_{i,j}$ for
$j^{\mathrm{th}}$ sample point to distinguish them from our model notation.

We assume zero mean Gaussian observation noise with $d\times d$ covariance
matrix $\boldsymbol{\Sigma}_{o,i,j}$ depending on the object index $i$ and the
sample point $j$. Assuming statistically independent noise along a single
trajectory, we form the observation noise covariance for $\mathbf{c}_{i}$ as a
$md\times md$ block diagonal matrix $\boldsymbol{\Sigma}_{o,i}$ where the
$j^{\mathrm{th}}$ diagonal block is given by $\boldsymbol{\Sigma}_{o,i,j}$.
Figure 1a illustrates the time and trajectory dependence of the observation
noise.

Using this notation, the posterior estimate of model parameters for a single
trajectory $\mathbf{c}_{i}$ is given in closed form [19, 232–234]:

$\displaystyle\boldsymbol{\Sigma}^{\mathrm{post}}_{\omega,i}=$
$\displaystyle(\boldsymbol{\Sigma}_{\omega}^{-1}+\boldsymbol{\Phi}_{i}\boldsymbol{\Sigma}_{o,i}^{-1}\boldsymbol{\Phi}_{i}^{\mathrm{T}})^{-1}$
(2) $\displaystyle\boldsymbol{\omega}^{\mathrm{post}}_{i}=$
$\displaystyle\boldsymbol{\Sigma}^{\mathrm{post}}_{\omega,i}\boldsymbol{\Phi}_{i}\boldsymbol{\Sigma}_{o,i}^{-1}\mathbf{c}^{\mathrm{ob}}_{i}$

With this improved parameter estimate from (2), we can then compute the
_representation error_ by measuring the average distance error (ADE) along
each trajectory for every measurement:

$\mathrm{ADE}=\frac{1}{Nm}\sum_{i}^{N}\sum_{j}^{m}||(\boldsymbol{\phi}^{\mathrm{T}}(\tau_{i,j})\otimes
I_{d})\boldsymbol{\omega}_{i}^{\mathrm{post}}-\mathbf{c}^{\mathrm{ob}}_{i,j}||_{2}$
(3)

Since the projection of the ADE onto the longitudinal and lateral direction of
motion are of particular interest for driving applications, we also calculate
these projections and denote them as ADElon and ADElat. The object heading is
either provided by the measurement (A2 and WO) or is inferred from a Rauch-
Tung-Striebel (RTS) smoothing of the data (A1, cf. Section V-A).

The above estimation of representation error requires the specification of
observation covariance $\boldsymbol{\Sigma}_{o}$, prior covariance
$\boldsymbol{\Sigma}_{\omega}$ and model complexity $n$ \- neither of which is
given. We now employ the Empirical Bayes Method to estimate all three
quantities.

##  IV Estimation of Observation Noise, Prior Parameters and Model Complexity
via Empirical Bayes

The Empirical Bayes approach [20] allows us to bootstrap prior distributions
over model parameters if many independent samples of the same phenomenon are
observed, such as the object trajectories in our datasets. The idea is to
formulate the likelihood of all observed trajectories $\mathbf{C}$ as a
function of the prior parameters alone. This can be achieved by marginalizing
the actual model parameters. Optimal prior parameters maximize the resulting,
so-called type-2, likelihood.

In an ideal world, the only prior parameter to be estimated would be the
covariance matrix $\boldsymbol{\Sigma}_{\boldsymbol{\omega}}$. The observation
noise covariance matrices $\boldsymbol{\Sigma}_{o,i,j}$ would be derived from
the ego vehicle’s sensor setup and given with the dataset. Unfortunately, the
$\boldsymbol{\Sigma}_{o,i,j}$ are not provided in any dataset and so we have
to reverse engineer, i.e. estimate, them from the data.

Clearly, we cannot estimate individual $\boldsymbol{\Sigma}_{o,i,j}$ for every
trajectory and every time-point. Instead, we provide a structured
parameterization in the form
$\boldsymbol{\Sigma}_{o,i,j}=\boldsymbol{\Sigma}_{o}(\theta)$.

Since ego trajectories and agent trajectories result from different sensor
setups, we also differentiate the parameterization of their noise models.

###  IV-A Observation Noise Covariance for Ego Trajectories

We model the observation noise covariance for ego trajectories in _world_
coordinates, assuming constant observation noise in $x$ and $y$ direction for
all sample points. The covariance for one measurement point is expressed as:

$\displaystyle\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{ego,world}}$
$\displaystyle=\boldsymbol{\Sigma}_{o}^{\mathrm{ego,world}}=\begin{bmatrix}\sigma^{2}_{\mathrm{x}}&\sigma_{\mathrm{xy}}\\\
\sigma_{\mathrm{xy}}&\sigma^{2}_{\mathrm{y}}\\\ \end{bmatrix}$ (4)

with
$\sigma^{2}_{\mathrm{x}}=\sigma^{2}_{\mathrm{y}}=\sigma^{2}_{\mathrm{diag}}$
and $\sigma_{\mathrm{xy}}=\sigma_{\mathrm{cov}}$. We can extend it to the
covariance of one complete ego trajectory via
$\boldsymbol{\Sigma}_{o,i}(\theta_{\mathrm{ego}})=\boldsymbol{I}_{m}\otimes\boldsymbol{\Sigma}_{o}^{\mathrm{ego,world}}$
and thus for all ego trajectories, we only have two parameters to estimate
$\theta_{\mathrm{ego}}=[\sigma_{\mathrm{diag}},\sigma_{\mathrm{cov}}]$.

###  IV-B Observation Noise Covariance for Agent Trajectories

Since the ego vehicles used in the datasets feature a suite of LIDAR sensors,
the observation uncertainty for agents is best expressed in _polar_
coordinates. We assume a constant angular resolution ($\sigma^{2}_{\alpha}$)
and variable distance resolution ($\sigma^{2}_{r,i,j}$):

$\displaystyle\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,polar}}$
$\displaystyle=\begin{bmatrix}\sigma^{2}_{r,i,j}&0\\\ 0&\sigma^{2}_{\alpha}\\\
\end{bmatrix}$ (5)

We model $\sigma_{r,i,j}^{2}$ as an increasing function of the measured
distance $r_{i,j}$ between agent and ego of the $j^{\mathrm{th}}$ sample point
($j\in[1,2,\ldots,m]$) as well as the parameters
$[\beta_{0},\beta_{1},\beta_{2}]\in\mathbb{R}^{+}$:

$\sigma^{2}_{r,i,j}=\beta_{0}+\beta_{1}r_{i,j}+\beta_{2}r_{i,j}^{2}$ (6)

Clearly, this is not perfect since we are using measurements to parameterize
the uncertainty of these very measurements. But it is the best approach we
have and it is safe to assume $\sigma^{2}_{r,i,j}$ varies only slightly within
the uncertainty of $r_{i,j}$. Next, we transform the observation covariance
from the $polar$ frame to the Cartesian _ego_ frame
$\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,ego}}$ based on [21, 77].

Finally, we rotate $\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,ego}}$ from the
_ego_ frame to the _world_ frame with:

$\displaystyle\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,world}}=\boldsymbol{R}_{i,j}(\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,ego}}+\sigma_{c}^{2}\boldsymbol{I}_{d})\boldsymbol{R}^{T}_{i,j}$
(7)

where $\boldsymbol{R}_{i,j}$ denotes the rotation matrix at $j^{\mathrm{th}}$
timestamp in the agent trajectory. The additional diagonal term
$\sigma_{c}^{2}$ is necessary to model errors resulting from timing
instability of the tracking system especially at close ranges.

The observation covariance
$\boldsymbol{\Sigma}_{o,i}(\theta_{\mathrm{agent}})$ for one complete agent
trajectory is then an $md\times md$ block diagonal matrix where the
$j^{\mathrm{th}}$ block is given by
$\boldsymbol{\Sigma}_{o,i,j}^{\mathrm{agent,world}}$. Hence, we have only 5
parameters to estimate:
$\theta_{\mathrm{agent}}=[\sigma_{\alpha},\beta_{0},\beta_{1},\beta_{2},\sigma_{c}]$.
We illustrate one example with our observation covariance model in Figure 1
with covariance marked as black ellipses.

###  IV-C Estimating Prior Parameters via Empirical Bayes

Finally, we are in the position to formulate the type-2 likelihood. Following
[19, 172–176], we obtain:

$\displaystyle
p(\mathbf{C}|\boldsymbol{\Sigma}_{o}(\theta),\boldsymbol{\Sigma}_{\omega})=$
$\displaystyle\prod_{i=1}^{N}\int\mathcal{N}(\mathbf{c}^{\mathrm{ob}}_{i}|\boldsymbol{\Phi}^{T}_{i}\boldsymbol{\omega},\boldsymbol{\Sigma}_{o,i}(\theta))$
(8)
$\displaystyle\times\mathcal{N}(\boldsymbol{\omega}|\boldsymbol{0},\boldsymbol{\Sigma}_{\omega})\mathrm{d}\boldsymbol{\omega}$
$\displaystyle=$
$\displaystyle\prod_{i=1}^{N}\mathcal{N}(\mathbf{c}^{\mathrm{ob}}_{i}|\boldsymbol{0},\boldsymbol{\Sigma}_{o,i}(\theta)+\boldsymbol{\Phi}^{T}_{i}\boldsymbol{\Sigma}_{w}\boldsymbol{\Phi}_{i})$

where $\mathbf{C}$ denotes all $N$ trajectories in the dataset. We maximize
the log of (8) with respect to $\boldsymbol{\Sigma}_{\omega}$ and $\theta$ for
ego and agent trajectories separately using gradient descent method [22].
These optima represent the prior parameters estimated from the dataset:

$\hat{\theta},\hat{\boldsymbol{\Sigma}}_{\omega}=\operatorname*{argmax}_{\theta,\boldsymbol{\Sigma}_{\omega}}\log(p(\mathbf{C}|\boldsymbol{\Sigma}_{o}(\theta),\boldsymbol{\Sigma}_{\omega}))$
(9)

For any model complexity $n$, we can thus estimate Bayesian optimal trajectory
representations from given data by plugging the estimated prior parameters
$\hat{\boldsymbol{\Sigma}}_{\omega}$ and
${\boldsymbol{\Sigma}}_{o,i}(\hat{\theta})$ into (2).

With increasing $n$ we will be able to achieve smaller ADE and estimate lower
observation noise at the expense of an increasing number of parameters, i.e.
we will start to overfit. We next find the optimal trade-off between data fit
and model complexity.

###  IV-D Estimating Optimal Model Complexity

The Akaike Information Criterion (AIC) [23] and Bayesian Information Criterion
(BIC) [24] characterize the score for a model in terms of how well it fits the
data, minus how complex the model is to define. AIC and BIC are defined as:

$\displaystyle\mathrm{AIC}$
$\displaystyle=\frac{\log(p(\mathbf{C}|\boldsymbol{\Sigma}_{o}(\theta),\boldsymbol{\Sigma}_{\omega}))}{N}-\mathrm{dof}(\theta,\boldsymbol{\Sigma}_{\omega})$
(10) $\displaystyle\mathrm{BIC}$
$\displaystyle=\frac{\log(p(\mathbf{C}|\boldsymbol{\Sigma}_{o}(\theta),\boldsymbol{\Sigma}_{\omega}))}{N}-\frac{\mathrm{dof}(\theta,\boldsymbol{\Sigma}_{\omega})}{2}\log(m)$

where
$\mathrm{dof}(\theta,\boldsymbol{\Sigma}_{\omega})=\mathrm{dof}(\theta)+(dn+d)(dn+d+1)/2$
denotes the degrees of freedom in the observation covariance and model
parameter covariance. For ego and agent trajectories,
$\mathrm{dof}(\theta_{\mathrm{ego}})=2$ and
$\mathrm{dof}(\theta_{\mathrm{agent}})=5$, respectively. A maximum of either
criterion as a function of $n$ indicates optimal trade-off between data-fit
and model complexity. In general, BIC penalizes model complexity higher and
tends to pick a simpler model.

##  V Experiments

###  V-A Preprocessing: Data Selection and Outlier Detection

Dataset characteristics are summarized in Table I. For comparability, we limit
our analysis to vehicle trajectories as this agent class is present in all
three datasets A1, A2 and WO. We analyze trajectories for time horizons
$T\in[3\mathrm{s},5\mathrm{s},8\mathrm{s}]$. For $T$ smaller than the maximal
observation horizon in the dataset, we select time windows of size $T$ in
strides of $1$s for A1 and randomly, one from each trajectory, for A2 and WO.
From the much larger set of agent trajectories in WO, we limit our analysis to
a random sample of 300k agent trajectories (without outlier) to reduce
computational cost.

We consider all samples that are within sensor range of the ego vehicle for
$T-0.5$s to $T+0.5$s in the analysis of time horizon $T$ trajectories but
discard all static trajectories with lengths $\leq 0.5$m as these can be
trivially represented with small error.

We notice a number of outliers in the data due to tracking loss and
inconsistent timing, i.e. objects are reported at physically impossible
locations for the given timestamps. To automatically detect and discard such
trajectories, we employ a Rauch-Tung-Striebel (RTS) smoothing of the data with
a simple double integrator based on [25, 48] and adjust the parameters
according to [25, 59]. We discard trajectories for which the RTS-Smoother
estimates positions more than $2$m away from measurement or the estimated
longitudinal acceleration (deceleration) exceeds a
$6\frac{\mathrm{m}}{\mathrm{s}^{2}}$ ($-10\frac{\mathrm{m}}{\mathrm{s}^{2}}$)
threshold. Table II gives an overview of the percentage of outliers in
datasets discarded for trajectories of time horizon $T=5$s.

We notice the timing issue in A1, where $T$ varies from $4.81$s to $25.64$s
for 50 sample points with $10$Hz [26]. The RTS-Smoother detects ego outliers
in A2 due to the unstable velocity estimation at the trajectory’s start or
end. In WO, we find quantities of outliers detected by RTS-Smoother primarily
because the agents leave the sensor range and their positions reset to
$(0,0)$.

TABLE II: Percentage of Outliers for $T=5$s Trajectory Datasets | A1 | A2 | WO
---|---|---|---
ego | time  |  22.81 |  0 |  0.02
static  |  23.95 |  20.66 |  25.41
out of view  |  0 |  0 |  0
RTS  |  0 |  1.18 |  0
total  |  42.95 |  21.84 |  25.42
agent | time  |  22.81 |  0 |  0.02
static  |  0 |  4.95 |  1.70
out of view  |  0 |  0 |  19.40
RTS  |  6.81 |  0.86 |  19.45
total  |  28.11 |  5.81 |  20.83
|  |  |  |

###  V-B Results

####  V-B1 Estimation of Observation Noise

Table III reports results for $T=5$s at the value of $n=\hat{n}$ that
maximizes AIC. As expected, ego trajectories exhibit much lower observation
noise than agent trajectories in all datasets. The agent trajectories in A1
are significantly noisier than in A2 and WO. WO provides data with the least
estimated observation noise due to its offline tracking algorithm [18]. As
model misspecification is upper bounded by the estimated observation noise,
the very low estimated observation noise indicates the high representation
quality.

####  V-B2 Model Complexity and Representation Error

Figure 2 shows Box-plots for the longitudinal and lateral ADE for agent
trajectory samples with $T\in[3\mathrm{s},5\mathrm{s},8\mathrm{s}]$. We
indicate the best model complexity according to AIC and BIC. Table IV gives
the numerical results for both ego and agent trajectories at the model
complexity $n=\hat{n}$ that maximizes AIC.

TABLE III: Estimated Observation Noise for $T=5$s at $\hat{n}$ maximizing AIC Datasets | A1 | A2 | WO
---|---|---|---
$\theta_{ego}$ |  $\sigma_{diag}$ [m]  |  0.024 |  0.012 |  0.008
$\sigma_{cov}$ $[m^{2}]$ |  2e-4 |  3e-6 |  -1e-7
$\theta_{agent}$ |  $\sigma_{\alpha}$ [rad]  |  1e-3 |  6e-4 |  3e-4
$\sigma_{c}$ [m]  |  0.161 |  0.044 |  0.017
$\sigma_{r}$ [m] $r=10\mathrm{m},20\mathrm{m},40\mathrm{m}$ |  0.128 |  0.055 |  0.019
|  0.176 |  0.062 |  0.027
|  0.246 |  0.085 |  0.042
|  |  |  |

Figure 2 and Table IV clearly show the that longer trajectories warrant higher
model complexities for their representation, but also that the benefits of
higher $n$ are diminishing beyond an optimal model $\hat{n}$ as indicated by
AIC or BIC. Overall, we see how simple linear models of moderate complexity
can represent trajectories with very high fidelity. E.g., the
$6^{\mathrm{th}}$ degree polynomial can approximate the 8-seconds agent
trajectories with $3.7$cm longitudinal and $1.6$cm lateral ADE in WO. However,
we also observe large deviations ($>1$m). Inspecting these samples, we find
they correspond to physically implausible measurements due to timing jitter or
tracking loss that are not excluded by the RTS-Smoother.

TABLE IV: Representation Error of ego and agent trajectories [m]
(Lon. Lat.)

| T [s] |  | Dataset
---|---|---|---
|  |  A1 |  A2 |  WO
ego | 3 |  $\hat{n}$ |  3 |  5 |  4
ADE  |  0.022 0.002 |  0.004 0.001 |  0.004 0.001
99.9%  |  0.173 0.120 |  0.094 0.012 |  0.066 0.021
5 |  $\hat{n}$ |  5 |  6 |  6
ADE  |  0.022 0.004 |  0.007 0.002 |  0.005 0.002
99.9%  |  0.157 0.111 |  0.143 0.030 |  0.071 0.025
8 |  $\hat{n}$ |  - |  7 |  7
ADE  |  - |  0.019 0.006 |  0.013 0.004
99.9%  |  - |  0.251 0.090 |  0.114 0.069
agent | 3 |  $\hat{n}$ |  2 |  5 |  4
ADE  |  0.185 0.060 |  0.019 0.005 |  0.016 0.005
99.9%  |  1.753 0.835 |  0.311 0.114 |  0.359 0.056
5 |  $\hat{n}$ |  3 |  5 |  5
ADE  |  0.191 0.065 |  0.051 0.016 |  0.018 0.006
99.9%  |  1.774 0.848 |  0.735 0.284 |  0.382 0.062
8 |  $\hat{n}$ |  - |  5 |  6
ADE  |  - |  0.093 0.033 |  0.037 0.016
99.9%  |  - |  1.115 0.461 |  0.483 0.186
|  |  |  |  |

  * •

$\hat{n}$ denotes the best polynomial degree according to AIC. $99.9\%$ means
the 99.9 percentile of the representation error.

Figure 2: The longitudinal (left) and lateral (right) distance error of models
for agent trajectories with $T\in[3\mathrm{s},5\mathrm{s},8\mathrm{s}]$. ”a,
b” denote the model complexity $n=\hat{n}$ that maximizes AIC and BIC,
respectively. The upper whisker denotes the 99.9% percentile.

####  V-B3 Representation Error vs. Prediction Error

Let us compare the price we pay for the bias introduced by our basis functions
to the associated benefits in computational efficiency.

Figure 3 compares the _representation error_ of our linear trajectory models
to the _prediction error_ over future trajectories of state-of-the-art
unbiased sequence based prediction methods. We compare to $minADE_{k}$, i.e.
the minimum average distance error over top $k$ most-likely predicted
trajectories. Naturally, any error is larger than the true measurement noise
level. If the latter is low, as in WO, the inherent bias of a linear
formulation is negligible in the prediction task. It is still significantly
lower than the $minADE_{k}$ of predictions in datasets, where the noise level
is likely higher such as A1.

On the other hand, we deem the computational benefits of linear models large.
For $n=5$, a trajectory is specified completely by six kinematic constraints
(position, velocity and acceleration) at start and end points. Since three
kinematic constraints at the start can be estimated entirely from past
observation, the prediction problem is in fact reduced to predicting the
kinematics at the end point. Hence, $50\%$ of the accuracy of trajectory
prediction is due to exact tracking of initial kinematics and $50\%$ is due to
accurate prediction of kinematics at the _end_ of the prediction horizon,
only.

Figure 3: The representation $ADE$ of polynomials at $\hat{n}$ maximizing AIC
and the prediction $minADE_{k}$ ($k=6$) of Wayformer[6] and MultiPath++[8] for
agent trajectories with $T\in[3\mathrm{s},5\mathrm{s},8\mathrm{s}]$. Latest
prediction results can be found at [27, 28].

##  VI Conclusion

Vehicle trajectories can be modelled with high fidelity over timescales
relevant for trajectory prediction by simple linear combinations of basis
functions. We have characterized the trade-off between model complexity and
representation error by an empirical analysis of several large public
datasets. Using an Empirical Bayes approach, we have estimated models for
observation noise and prior distributions over model parameters. The estimated
observation noise parameters can (and should) be considered when training
trajectory prediction models with a Gaussian Log-Likelihood loss, particularly
when combining different datasets. The prior parameters can inform the motion
models of trajectory tracking and filtering models [15] or regularize
trajectory prediction models. The representation error of a linear formulation
is small compared to prediction errors of current state-of-the-art models.
This suggests that the bias inherent in linear models is much smaller than the
epistemic uncertainty in the prediction task. The computational benefits, on
the other hand, are large. Using linear models of vehicle trajectories, the
problem of predicting a trajectory over a finite time horizon $T$ can reduce
to predicting the kinematic end state at $t+T$, alone.

## ACKNOWLEDGMENT

Authors would like to thank Andreas Philipp for many useful discussions and
acknowledge funding from the German Federal Ministry for Economic Affairs and
Climate Action within the project “KI Wissen – Automotive AI powered by
Knowledge”.

## References

  * [1] Mayank Bansal, Alex Krizhevsky and Abhijit Ogale  “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst”  In _arXiv preprint arXiv:1812.03079_ , 2018
  * [2] Henggang Cui et al.  “Multimodal trajectory predictions for autonomous driving using deep convolutional networks”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 2090–2096  IEEE
  * [3] Jiyang Gao et al.  “Vectornet: Encoding hd maps and agent dynamics from vectorized representation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11525–11533
  * [4] Tung Phan-Minh et al.  “Covernet: Multimodal behavior prediction using trajectory sets”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14074–14083
  * [5] Ming Liang et al.  “Learning lane graph representations for motion forecasting”  In _European Conference on Computer Vision_ , 2020, pp. 541–556  Springer
  * [6] Nigamaa Nayakanti et al.  “Wayformer: Motion Forecasting via Simple & Efficient Attention Networks”  In _arXiv preprint arXiv:2207.05844_ , 2022
  * [7] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty and Marco Pavone  “Trajectron++: Dynamically-Feasible Trajectory Forecasting with Heterogeneous Data”  In _Computer Vision – ECCV 2020_ , 2020, pp. 683–700
  * [8] Balakrishnan Varadarajan et al.  “Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 7814–7821  IEEE
  * [9] Charles C Macadam  “Understanding and modeling the human driver”  In _Vehicle system dynamics_ 40.1-3 Taylor & Francis, 2003, pp. 101–134
  * [10] Hasti Hayati, David Eager, Ann-Marie Pendrill and Hans Alberg  “Jerk within the context of science and engineering—A systematic review”  In _Vibration_ 3.4 MDPI, 2020, pp. 371–409
  * [11] Il Bae et al.  “Self-driving like a human driver instead of a robocar: Personalized comfortable driving experience for autonomous vehicles”  In _arXiv preprint arXiv:2001.03908_ , 2020
  * [12] Thibault Buhet, Emilie Wirbel, Andrei Bursuc and Xavier Perrotton  “Plop: Probabilistic polynomial objects trajectory planning for autonomous driving”  In _arXiv preprint arXiv:2003.08744_ , 2020
  * [13] Zhaoen Su et al.  “Temporally-Continuous Probabilistic Prediction using Polynomial Trajectory Parameterization”  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 3837–3843  IEEE
  * [14] Mohammadhossein Bahari et al.  “Vehicle trajectory prediction works, but not everywhere”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17123–17133
  * [15] Jörg Reichardt  “Trajectories as Markov-States for Long Term Traffic Scene Prediction”  In _14-th UniDAS FAS-Workshop_ , 2022, pp. 14
  * [16] Ming-Fang Chang et al.  “Argoverse: 3D Tracking and Forecasting With Rich Maps”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019
  * [17] Benjamin Wilson et al.  “Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting”  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021
  * [18] Scott Ettinger et al.  “Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9710–9719
  * [19] Kevin P Murphy  “Machine learning: a probabilistic perspective”  MIT press, 2012
  * [20] Bradley Efron  “Large-scale inference: empirical Bayes methods for estimation, testing, and prediction”  Cambridge University Press, 2012
  * [21] Nico Kämpchen  “Feature-level fusion of laser scanner and video data for advanced driver assistance systems”, 2007
  * [22] Joshua V Dillon et al.  “Tensorflow distributions”  In _arXiv preprint arXiv:1711.10604_ , 2017
  * [23] Hirotugu Akaike  “Information theory and an extension of the maximum likelihood principle,[w:] proceedings of the 2nd international symposium on information, bn petrow, f”  In _Czaki, Akademiai Kiado, Budapest_ , 1973
  * [24] Gideon Schwarz  “Estimating the dimension of a model”  In _The annals of statistics_ JSTOR, 1978, pp. 461–464
  * [25] Andreas Philipp  “Perception and Prediction of Urban Traffic Scenarios for Autonomous Driving”, 2021
  * [26] , https://github.com/argoai/argoverse-api/issues/124
  * [27] , https://eval.ai/web/challenges/challenge-page/454/leaderboard/1279
  * [28] , https://waymo.com/open/challenges/2022/motion-prediction