# Informed Priors for Knowledge Integration in Trajectory Prediction

Christian Schlauch
Humboldt-Universität zu Berlin,
Chair of Statistics and Data Science
and Continental AG
Berlin, Germany
& Nadja Klein
Humboldt-Universität zu Berlin
Chair of Statistics and Data Science
Berlin, Germany & Christian Wirth
Continental AG
Frankfurt am Main, Germany

###### Abstract

Informed machine learning methods allow the integration of prior knowledge
into learning systems. This can increase accuracy and robustness or reduce
data needs. However, existing methods often assume hard constraining
knowledge, that does not require to trade-off prior knowledge with
observations, but can be used to directly reduce the problem space. Other
approaches use specific, architectural changes as representation of prior
knowledge, limiting applicability. We propose an informed machine learning
method, based on continual learning. This allows the integration of arbitrary,
prior knowledge, potentially from multiple sources, and does not require
specific architectures. Furthermore, our approach enables probabilistic and
multi-modal predictions, that can improve predictive accuracy and robustness.
We exemplify our approach by applying it to a state-of-the-art trajectory
predictor for autonomous driving. This domain is especially dependent on
informed learning approaches, as it is subject to an overwhelming large
variety of possible environments and very rare events, while requiring robust
and accurate predictions. We evaluate our model on a commonly used benchmark
dataset, only using data already available in a conventional setup. We show
that our method outperforms both non-informed and informed learning methods,
that are often used in the literature. Furthermore, we are able to compete
with a conventional baseline, even using half as many observation examples.

_Keywords_ Continual Learning $\cdot$ Informed Learning $\cdot$ Prior
Knowledge $\cdot$ Trajectory Prediction $\cdot$ Drivable Area Compliance
$\cdot$ Autonomous Driving

##  1 Introduction

Deep learning approaches can achieve excellent prediction performance in many
domains, but require large training data sets and can be brittle in open world
applications (Wörmann et al.,, 2022). However, training data may be difficult
to obtain, e.g. because relevant events are too rare or too costly to observe.
Informed machine learning (von Rueden et al.,, 2021) tries to alleviate this
problem by integrating prior knowledge. This means, physical, world or expert
knowledge is explicitly integrated into the learning system, and not just
implicitly via observations. Doing so, allows to reduce the required training
data, or increase accuracy, because less information needs to be derived from
observations. Furthermore, the system can be more robust and safe, especially
when encountering unseen events. (Wörmann et al.,, 2022)

Such an approach is of particular interest in the domain of autonomous
driving, where applications have to reliably work under a wide variety of
environments and safety-critical events, like near-collisions (Wörmann et
al.,, 2022). Furthermore, prior knowledge can be substantially cheaper in this
domain, as it can be potentially defined with only map data (Boulton et al.,,
2021). An exemplary application is the trajectory prediction problem, that is
especially difficult, since probabilities for multiple outcomes have to be
predicted (Huang et al.,, 2022).

We contribute to the area of informed machine learning and propose a
probabilistic informed learning method based on continual learning (De Lange
et al.,, 2022). The basic idea is to start with a learning task, that captures
the prior knowledge. Then, in a second task when observed data is available,
this prior knowledge can be turned into a weight posterior distribution, which
can then be used as informed prior for the observation-based learning in
future tasks. Such an approach has multiple advantages: (1) It is possible to
integrate arbitrary prior knowledge, not just scientific knowledge, such as
physical limits (Cui et al.,, 2020; Li et al.,, 2019). (2) We do not assume a
specific representation of knowledge, only that it can be mapped to a
prediction target. (3) The probabilistic formulation allows us to infer
complete predictive distributions and achieve calibrated, multi-modal outputs
beyond simple point predictions. Probabilistic formulations have been shown to
lead to improved models, even if calibration and multi-modality only play a
minor role (McAllister et al.,, 2017). (4) Furthermore, the approach allows us
to potentially integrate multiple sources of prior information by learning
multiple priors or by updating the prior sequentially.

As an exemplary illustration, we consider the trajectory prediction and
present a concrete implementation by adapting the state-of-the-art CoverNet
trajectory predictor (Phan-Minh et al.,, 2020) with generalized variational
continual learning (GVCL; Loo et al.,, 2021) and by using prior knowledge
about the drivable area (Boulton et al.,, 2021).

In multiple sets of experiments, we critically analyze our approach and show
that it improves on existing non-informed baselines and other informed
learning approaches like multi-task or transfer learning (Boulton et al.,,
2021).

In summary, our contributions are:

  * $\bullet$

A novel approach for integrating prior knowledge as soft constraint into deep
learning models, that can also deal with multi-modal and probabilistic
prediction problems, distinct datasets and does not require architectural
changes;

  * $\bullet$

an implementation that substantially improves the CoverNet trajectory
predictor, only using data already available in the conventional setup;

  * $\bullet$

a demonstration of the advantages of our informed machine learning method in
the domain of trajectory prediction for autonomous driving;

  * $\bullet$

an empirical assessment of the different facets of our probabilistic approach.

The paper is structured as follows: In Sec. 2 we introduce our informed
machine learning approach before we exemplify it on the CoverNet architecture
in Sec. 3. Related work is covered in Sec. 4. In Sec. 5 we benchmark our
method against various baseline models and close the paper with a conclusion
and discussion in Sec. 6.

##  2 Continual Learning with Prior Knowledge Integration

We introduce a regularization-based continual learning approach to explicitly
integrate prior knowledge in the learning algorithm, aiming to increase the
accuracy, robustness and safety of deep learning models, while making minimal
assumptions on the type of prior knowledge or model architecture. In
particular, we consider world and expert knowledge as defined in von Rueden et
al., (2021) as our prior knowledge sources. In contrast to scientific
knowledge, this knowledge can be violated and it is required to trade-off in
form of soft constraints.

###  2.1 The Informed Learning Problem

Our focus is on supervised learning problems, where we denote
$\mathcal{D}=\\{\mathcal{X}\times\mathcal{Y}\\}$ as the domain over the input
space $\mathcal{X}$ and the output space $\mathcal{Y}$. From this, we draw a
labelled dataset $D=\\{(x_{i},y_{i})\\}^{N}_{i=1}$ consisting of a finite
number of $N$ conditionally independent realizations $y_{i}|x_{i}$, where
$x_{i}\in\mathcal{X}$ and $y_{i}\in\mathcal{Y}$. Let $p(y|x)$ denote the true
conditional distribution of $Y$ given $X=x$ and let $\mathcal{H}$ denote a
hypothesis space consisting of hypothesis (models) $h_{\theta}(\cdot)$ that
are defined through some model parameters $\theta$. Given $D$ and
$\mathcal{H}$, a learning algorithm $\mathcal{L}:D\to\mathcal{H}$ is applied
to learn a final hypothesis $h_{\theta}(x)\in\mathcal{H}$, either in form of a
predictive function $y=h_{\theta}(x)$ or in a probabilistic approach in form
of a predictive distribution $h_{\theta}(x)\sim p_{\theta}(y|x)$. The goal is
now to integrate the prior knowledge source $\mathcal{B}$ in possibly any of
the elements $D$, $\mathcal{L}$ and $\mathcal{H}$ or in revision of
$h_{\theta}(x)$ itself, thereby placing an inductive bias into the learning
problem that increases accuracy, robustness, parsimony or compliance (Wörmann
et al.,, 2022; von Rueden et al.,, 2021). Following von Rueden et al., (2021),
we call the explicit integration of a prior knowledge source $\mathcal{B}$ an
_informed learning problem_.

###  2.2 Informed Priors with Continual Learning

Figure 1: Example for the integration of prior knowledge with continual
learning: Using the drivable area as informed prior for trajectory prediction.

In a continual learning scenario we generally assume to sequentially encounter
$t=1,\ldots,T$ disjoint, labelled training datasets (tasks)
$D_{t}=\\{(x^{(t)}_{i},y^{(t)}_{i})\\}^{N_{t}}_{i=1}$ with $N_{t}$ independent
samples each. These datasets may potentially have been drawn from $T$
different domains $\mathcal{D}_{t}$ with different input spaces
$\mathcal{X}_{t}$, output spaces $\mathcal{Y}_{t}$ under distinct conditional
probability distributions $p_{t}(y|x)$ (Parisi et al.,, 2019; De Lange et
al.,, 2022). However, we assume that each dataset represents a related task
and the general goal is to learn a single model that is able to predict well
when seeing data from any of them (De Lange et al.,, 2022). In particular, we
want to consider a probabilistic deep learning model with parameters $\theta$
that returns a predictive distribution $p_{\theta}(y|x)$ over an output $y$
given an input $x$. Placing a prior distribution $\pi(\theta)$ on $\theta$
allows to recursively apply Bayes’ rule over all tasks given each likelihood
$p_{\theta}(\cdot|x)$ to recover the posterior probability distributions
$p(\theta|D_{1:T})$ with

$\begin{split}p(\theta|D_{1:T})\propto\pi_{0}(\theta)\prod^{T}_{t=1}p_{\theta}(y_{t}|x_{t}).\end{split}$
(1)

and $\pi_{0}(\theta)$ as the initial, uniformed prior. As uninformed prior, we
use a zero-mean, unit-variance normal distribution.

Applying this approach to an informed learning problem, we assume that a given
prior knowledge source $\mathcal{B}$ can be represented in form of a dataset
$D_{\mathcal{B}}$. We can then define a task $t=1$ to learn the corresponding
posterior distribution $\pi_{\mathcal{B}}(\theta)$ which constitutes an
implicit probabilistic representation of $\mathcal{B}$. Now, given an
observational dataset $D_{\mathcal{O}}$ in a second task $t=2$, we can use
$\pi_{\mathcal{B}}(\theta)$ as an informed prior and regularize the posterior
distribution $p(\theta|D_{\mathcal{O}})$ as
$p_{\theta}(y_{\mathcal{O}}|x_{\mathcal{O}})\pi_{\mathcal{B}}(\theta)$. We
present an exemplary visualization of this approach in Fig. 1 for the
trajectory prediction in autonomous driving considered later.

This regularization-based approach implicitly trades-off the effect of
$\pi_{\mathcal{B}}(\theta)$ with the available data in $D_{\mathcal{O}}$,
making it suitable to consider world or expert knowledge as soft constraint.
The less observational data is available, e.g. for rare events in the input
space $\mathcal{X}$, the more relevant the informed prior becomes. In theory,
we may consider even multiple knowledge sources $\mathcal{B}_{t}$
sequentially, constructing a knowledge pipeline. The separation of concerns in
such a sequential approach allows to reuse posteriors
$\pi_{D_{1:t-1}}(\theta)$ as informed priors of the pipeline, to expand the
latter when new data becomes available or to reorder the tasks to form a
curriculum (Mundt et al.,, 2020).

In general, this approach does not require a specific model representation and
can be applied to existing state-of-the-art models using a shared output
representation for all tasks. We also note that the assumed dataset
$D_{\mathcal{B}}$ can be derived from different knowledge representations,
e.g. by generating artificial examples in a simulation or by making semantic
annotations in an observational dataset. In theory, due to the implicit trade-
off in the approach, the number of samples required in $D_{\mathcal{B}}$ does
not necessarily have to stand in balanced relation to the number of samples in
$D_{\mathcal{O}}$.

###  2.3 Societal Impact

In many domains, we are confronted with implicit, societal biases in the
training data; e.g. we may observe drastic differences in driver’s behavior
between countries. In theory, our approach can be used to adapt a model by
integrating available prior knowledge about social norms or regulations that
determine such behavior. Furthermore, our approach may be used to validate a
given model by testing its performance in a prior knowledge task, e.g. to
check whether it complies to certain regulations or exhibits certain social
norms or biases. We presume that such validation may become necessary for the
certification of e.g. autonomous driving systems in the conceivable future.
Therefore, our approach may have a positive impact. However, like any
automatization efforts, advances in autonomous driving may lead to a reduction
of available jobs.

##  3 Generalized Variational Continual Learning-CoverNet for Trajectory
Prediction

Figure 2: CoverNet architecture from Phan-Minh et al., (2020).

We exemplify our continual learning approach by applying it to CoverNet (Phan-
Minh et al.,, 2020), a state-of-the-art trajectory prediction model (Liu et
al.,, 2021; Huang et al.,, 2022). We use drivable area map data as a prior
knowledge source, which is also applied by Boulton et al., (2021) and allows
comparisons to existing baselines. The trajectory prediction in autonomous
driving is a safety-critical application (Wörmann et al.,, 2022; Liu et al.,,
2021) that renders the use of probabilistic multi-modal predictions useful to
obtain calibrated probabilities along with accurate uncertainty measures for
safety-critical events.

###  3.1 The Trajectory Prediction Problem

Consider a scene on the road, where a self-driving system is assumed to
observe states $s_{i}$ in the state space $\mathcal{S}$ of all involved agents
$a_{i}\in\mathcal{A}$. Let $s_{i}^{(\tau)}\in\mathcal{S}$ denote the state of
agent $a_{i}\in\mathcal{A}$ at time $\tau$ and let
$s_{i}^{(\tau-\mathrm{T}_{o}:\tau)}=\\{s_{i}^{(\tau-\mathrm{T}_{o})},s_{i}^{(\tau-\mathrm{T}_{o}+1)},\ldots,s_{i}^{(\tau)}\\}$
be its trajectory over an observation period $\mathrm{T}_{o}$. Additionally,
we assume access to map data $M\in\mathcal{M}$ and some available prior
knowledge $B\in\mathcal{B}$ that can be translated into a prior
$\pi_{\mathcal{B}}(\theta)$. The map data itself may include semantic
information, e.g. road markings and traffic signals, that can be used as
further prior knowledge source. The scene context is then denoted as
$\mathcal{C}=\\{\\{s_{i}^{((\tau-\mathrm{T}_{o}):\tau)}\\}^{|\mathcal{A}|}_{i=1},M,B\\}$.
Given $\mathcal{C}$, the goal is to predict the distribution of future
trajectories $p(s_{i}^{(\tau:(\tau+\mathrm{T}_{h}))}|\mathcal{C})$ over the
prediction horizon $\mathrm{T}_{h}$.

###  3.2 CoverNet informed by Driveable Area Knowledge

CoverNet as introduced by Phan-Minh et al., (2020) solves the trajectory
prediction problem, using a birds-eye-view RGB rendering that combines map
data and states of all involved agents over a three second observation period.
The output is represented by sparse trajectory sets $\mathcal{K}(\epsilon)$
that approximate $p(s_{i}^{(\tau:(\tau+\mathrm{T}_{h}))}|\mathcal{C})$ by a
predefined set of possible trajectories. The sparsity is controlled by a
coverage bound $\epsilon$ in meter, which determines the maximum distance
between two trajectories in the set and thus, the total number of elements
$|\mathcal{K}(\epsilon)|$ in the set. Phan-Minh et al., (2020) computes
$\mathcal{K}(\epsilon)$ by clustering all observed trajectories in a given
dataset, with a cluster distance of $\epsilon$. This output representation
allows to treat each trajectory in the set as a sample of the predictive
distribution, while only computing its conditional probability. This allows
the authors to reduce the task to a classification problem, with the goal to
approximate the probability of each class (or trajectory distribution sample).
The architecture is visualized in Fig. 2.

We implement our approach in Sec. 2 by applying generalized variation
continual learning (GVCL; Loo et al.,, 2021) to CoverNet. GVCL is a
likelihood-tempered version of variational continual learning (VCL; Nguyen et
al.,, 2018) based on variational inference (VI; Wilson and Izmailov,, 2020).
The likelihood-tempering has been shown to help in VI (Wilson and Izmailov,,
2020), in some settings. GVCL solves the recursion described in Eq. (1) by
minimizing the likelihood-tempered log-evidence lower bound (so-called
$\beta$-ELBO, $\beta\in[0,1]$) at every task $t=1,\ldots,T$. This yields a
tractable approximate solution $q_{t}(\theta)\approx p(\theta|D_{1:t})$ using
the approximate posterior $q_{t-1}(\theta)\approx p(\theta|D_{1:t-1})$ of the
previous task $t-1$ as a prior distribution weighted by $\beta$ for all $t$.
More precisely, the $\beta$-ELBO at time $t$ is defined by

$\displaystyle\beta\text{-ELBO}=\mathbb{E}_{\theta\sim
q_{t}(\theta)}\left[\log{p_{\theta}(y_{t}|x_{t})}\right]-\beta
D_{\text{KL}}(q_{t}(\theta)\|q_{t-1}(\theta)),$ (2)

where $D_{\text{KL}}(q_{1}||q_{2})$ is the Kullback-Leibler (KL) divergence
between two probability distributions $q_{1},q_{1}$. We assume a fixed form
$q_{t}(\theta)$ that belongs to the family $\mathcal{Q}$ of Gaussian
distributions with diagonal covariance matrices. In our GVCL-CoverNet
architecture, $\theta$ collects as usually all weights and biases, and the
variational parameter vector $\nu=(\mu,\sigma)^{\top}$ consists of the means
and standard deviations of the independent Gaussian variational distributions
of each model weight.

We inform the trajectory prediction by placing a soft constraint on on-road
trajectories. Thus, in the first task we assume that drivable area map data is
available in the scene context $\mathcal{C}$. Given this map data, we define a
multi-label classification where the positive labels are given by a binary
vector $r_{i}^{(\tau)}\in\mathbb{R}^{|\mathcal{K}|}$ for an agent $a_{i}$ at
time $\tau$, defined such that its value is 1 at entry $k_{i,j}^{(\tau)}$ if
the $j$th tracetory $k_{i,j}^{(\tau)}$ in the set $\mathcal{K}$ is entirely
contained in the drivable area and 0 otherwise. The model is then trained
using a binary cross-entropy loss. The resulting posterior is used in the
second task to regularize the model according to Eq. (2). Given the
observational dataset $D_{\mathcal{O}}$, this task is defined as a multi-class
classification with positive samples determined by the element in the
trajectory set closest to the actual ground truth. The model is finally
trained using a sparse categorical cross-entropy loss where its logit
transformations are normalized by the softmax function, inline with the
original CoverNet algorithm.

Note, even though we cannot enforce the prior knowledge as hard constraint, we
may artificially sharpen the informed prior in the $D_{\text{KL}}$-term as
described by Loo et al., (2021). This allows to strengthen the effect of the
informed prior, making GVCL a particularly flexible method choice.
Furthermore, GVCL can be applied to wide variety of model architectures in
general (Nguyen et al.,, 2018; Loo et al.,, 2021), with the exception of
recurrent architectures for which variational reformulations are typically
more involved (Gal and Ghahramani,, 2016).

##  4 Related Work

We briefly embed our contribution into the literature by reviewing related
work next. A general survey on informed machine learning can be found in von
Rueden et al., (2021) providing a taxonomy based on the type of knowledge
source, representation and integration. Prior knowledge integration in the
domain of autonomous driving is reviewed by e.g. Wörmann et al., (2022).

In the trajectory prediction for autonomous driving, there has been a
particular focus on integrating scientific knowledge about physical
limitations using plausible dynamical models. For instance, Phan-Minh et al.,
(2020) propose dynamically constructed trajectory sets as output
representation, while Cui et al., (2020) and Li et al., (2019) define specific
layer types to directly encode the dynamical model. In contrast, Bahari et
al., (2021) present a post-processing approach based on a model predictive
controller. While these methods enforce the physical limitations as hard
constraints, we enable the use of less formalized world and expert knowledge
as soft constraint. Earlier work in this direction has been done by Wang et
al., (2020) using a multi-task approach, while Boulton et al., (2021)
introduce a loss based method combined with a transfer learning approach where
the model is pre-trained using prior knowledge. Similar approaches have also
been shown to be beneficial in other domains (von Rueden et al.,, 2022).
However, transfer learning is typically limited to an unidirectional knowledge
transfer between two tasks, while multi-task learning and loss approaches
require a single dataset with simultaneously available labels. A continual
learning approach can be applied without these limitations. To our knowledge,
a continual learning approach has not yet been used to realize an informed
machine learning algorithm.

Continual learning methods are typically categorized as regularization-based,
rehearsal-based or as architectural (Parisi et al.,, 2019; De Lange et al.,,
2022). Architectural methods typically require a specific model
representation, while rehearsal-based methods require a so-called replay
buffer. Regularization-based methods do not have these requirements and can be
easily formulated in a probabilistic way, which is itself argued to feature
robustness and safety (McAllister et al.,, 2017). We use the VI-based GVCL
method, which generalizes variational continual learning (VCL; Nguyen et al.,,
2018) and online elastic weight consolidation (Online EWC; Kirkpatrick et
al.,, 2017; Schwarz et al.,, 2018).

##  5 Experiments

We empirically investigate the performance of the proposed GVCL-CoverNet model
using the NuScenes dataset of Caesar et al., (2020). This dataset contains
1000 scenes, each being 20 seconds, sampled at a frequency $F=2\text{ Hz}$. In
the following, we introduce two non-informed and two informed baseline methods
as competitors. Thereafter, we present the experimental design, the
implementation details and lastly the results.

###  5.1 Baseline Models

We consider two non-informed baseline models:

  * $\bullet$

Base-CoverNet, as introduced by Phan-Minh et al., (2020) and reviewed in Sec.
3. All other models used in our experiments are adaptations of this model.

  * $\bullet$

VI-CoverNet is a variational adaptation of the Base-CoverNet model. Like our
GVCL-CoverNet model described in Sec. 3, the number of parameters is doubled,
in comparison to Base-CoverNet, and the likelihood-temperature is set by a
hyperparameter $\beta\in[0,1]$. The consideration of this baseline allows us
to distinguish the effects of the continual learning approach and the
variational formulation alone.

We also consider two informed baseline models that enforce on-road trajectory
predictions as soft constraint:

  * $\bullet$

Loss-CoverNet uses an auxiliary off-road loss as introduced by Boulton et al.,
(2021). The model jointly trains on the prior knowledge and the observations.
The trade-off between both losses is controlled by a hyperparameter
$\lambda_{\text{{multi}}}>0$.

  * $\bullet$

Transfer-CoverNet uses a model pre-trained using prior knowledge, similar to
Boulton et al., (2021).

Since our GVCL-CoverNet model has mean and scale as variational parameters, we
also introduce GVCl-Det-CoverNet to allow for a fair comparison to the non-
variational baselines at test time. The GVCL-Det-CoverNet is still trained
according to the GVCL method introduced in Sec. 3 but makes non-probabilistic
predictions at test time using the mean $\mu$ of $\nu$ only.

###  5.2 Experimental Design

In total, we run three different sets of experiments with 10%, 50% and 100% of
the available observations in the training set. However, we use all available
scenes for determining the drivable area prior. This relates to real world
driving applications, where map data is substantially cheaper to obtain than
observations and where thus prior knowledge is cheaper to assess than
observations.

We use the data split described by Phan-Minh et al., (2020) with 32186
observations in the full training set, 8560 observations in the validation set
and 9041 observations in the test set. The 10% and 50% subsets are randomly
sub-sampled from the training data, while using the full validation and test
set.

In our first set of experiments, we use the fixed trajectory sets
$\mathcal{K}(\epsilon)$ from the NuScenes prediction challenge (Caesar et
al.,, 2020) with the coverage bound $\epsilon=4$, corresponding to
$|\mathcal{K}(4)|=415$ modes. In a second part, we investigate the influence
of the coverage bound and additionally consider the bounds
$\epsilon\in\\{2,8\\}$, corresponding to $|\mathcal{K}(2)|=2206$ and
$|\mathcal{K}(8)|=64$ modes, respectively.

To evaluate each experiment, we consider the following metrics that have been
commonly applied in the literature (Huang et al.,, 2022; Liu et al.,, 2021).
The average displacement error minADE1 and final displacement error minFDE1
evaluate the quality of the most likely trajectory, whereas minADE5 considers
the five most likely trajectories. Please note, that these metrics are limited
by the resolution of the trajectory set $\mathcal{K}$. Therefore, we also
report the rank (RNK), determining the position of the best trajectory, when
ranking the predictions by probability. Top-1 accuracy (ACC) determines how
often the best trajectory from the trajectory set is predicted with the
highest probability. The negative log likelihood (NLL) and expected
calibration error (ECE) allow to evaluate the calibration of the probabilistic
multi-class prediction. The drivable area compliance (DAC) metric determines
in how far the predictions conform to the prior knowledge. The minADE5 and the
RNK metric indirectly depend on well calibrated, probabilistic predictions, as
they evaluate the quality of the probability-based ordering.

###  5.3 Implementation Details

All presented models have the same architecture, as described by Phan-Minh et
al., (2020). However, we do not use ImageNet pre-training, as we do not
observe any improvements on the baseline model and there are no ImageNet
weights readily available for the variational models. Instead, we initialize
the deterministic weights $\theta$ and variational means as proposed by He et
al., (2015). The variational standard deviations $\sigma$ are drawn from the
Gaussian distribution $\mathcal{N}(0.005,0.1)$ truncated at $(0.005,0.205)$.
The deterministic models are trained with stochastic gradient descent (SGD;
Robbins,, 2007) over 20 epochs. The variational models are trained by
minimzing the respective ELBO with the reparameterized gradient method (Kingma
and Welling,, 2014) over 2000 epochs and linearly decaying learning rate. We
slightly temper the posterior of the variational models by setting $\beta$ to
1 divided by batch size, following Wilson and Izmailov, (2020). At test time,
the variational models average over 7 samples.

We individually tune the learning rates and batch sizes for each model as well
as the $\lambda_{\text{{multi}}}$ for the Loss-CoverNet baseline model using
100% of the training data and coverage bounds $\epsilon\in\\{2,8\\}$. We adopt
the tuning results for $\epsilon=8$ for the experiments with $\epsilon=4$. The
values are selected to minimize the validation NLL, using the heteroscedastic
and evolutionary Bayesian optimizer (HEBO Cowen-Rivers et al.,, 2022). The
chosen values are summarized in Tab. 1.

The models have been implemented in Tensorflow (Abadi et al.,, 2016),
published on Github111Anonymized. The experiments have been executed on a high
performance computation cluster with 4 NVIDIA GeForce RTX 3090-24GB GPUs and
NVIDIA V100-32GB GPUs with parallelization for 4 GPUs.

Table 1: Selected values for tuned hyperparameters. Method | Batch Size | Learning Rate | $\lambda_{{\text{multi}}}$
---|---|---|---
$\epsilon$=2 |  |  |
Base | 5 | 0.00083 |
Transfer | 5 | 0.00083 |
Loss | 5 | 0.00083 | 0.01
VI | 12 | 0.003 |
GVCL | 12 | 0.003 |
$\epsilon$=4 |  |  |
Base | 16 | 0.0008 |
Transfer | 16 | 0.0002 |
Loss | 16 | 0.0001 | 0.01
VI | 12 | 0.003 |
GVCL | 12 | 0.003 |
$\epsilon$=8 |  |  |
Base | 16 | 0.0008 |
Transfer | 16 | 0.0002 |
Loss | 16 | 0.0001 | 0.01
VI | 12 | 0.003 |
GVCL | 12 | 0.003 |
Table 2: Results for constrained observational training datasets with coverage bound $\epsilon=4$ (bold as unique best result and italic as unique best deterministic per set) Experiment |  ACC |  NLL |  ECE |  RNK |  minADE1 |  minADE5 |  minFDE1 |  DAC
---|---|---|---|---|---|---|---|---
100% Data |  |  |  |  |  |  |  |
Base |  0.15 |  3.40 |  0.05 |  15.07 |  4.77 |  2.26 |  10,61 |  0.88
Transfer |  0.15 |  3.24 |  0.06 |  11.56 |  4.70 |  2.12 |  10.15 |  0.96
Loss |  0.13 |  3.58 |  0.13 |  16.85 |  4.88 |  2.40 |  10.53 |  0.94
VI |  0.19 |  3.14 |  0.03 |  11.07 |  4.13 |  2.03 |  9.21 |  0.91
GVCL |  0.19 |  2.98 |  0.02 |  9.51 |  3.99 |  1.94 |  8.96 |  0.94
GVCL-Det |  0.15 |  3.60 |  0.18 |  11.85 |  4.55 |  2.26 |  9.93 |  0.91
50% Data |  |  |  |  |  |  |  |
Base |  0.12 |  3.72 |  0.07 |  20.88 |  5.47 |  2.60 |  12.03 |  0.87
Transfer |  0.13 |  3.55 |  0.07 |  15.77 |  5.24 |  2.33 |  11.14 |  0.96
Loss |  0.11 |  3.90 |  0.05 |  24.83 |  5.84 |  2.68 |  12.61 |  0.91
VI |  0.16 |  3.53 |  0.04 |  16.49 |  4.61 |  2.21 |  10.24 |  0.89
GVCL |  0.18 |  3.20 |  0.02 |  12.67 |  4.30 |  1.63 |  9.57 |  0.92
GVCL-Det |  0.14 |  3.75 |  0.13 |  15.67 |  4.76 |  2.41 |  10.41 |  0.91
10% Data |  |  |  |  |  |  |  |
Base |  0.09 |  4.46 |  0.06 |  41.37 |  6.37 |  3.06 |  13.81 |  0.85
Transfer |  0.08 |  4.36 |  0.12 |  32.84 |  6.41 |  3.14 |  13.41 |  0.98
Loss |  0.08 |  4.48 |  0.07 |  39.51 |  6.77 |  3.24 |  14.25 |  0.84
VI |  0.10 |  4.53 |  0.07 |  35.61 |  6.03 |  2.80 |  12.77 |  0.82
GVCL |  0.13 |  3.86 |  0.03 |  27.73 |  5.51 |  2.55 |  11.86 |  0.91
GVCL-Det |  0.11 |  4.16 |  0.13 |  25.66 |  5.77 |  2.75 |  12.08 |  0.93

###  5.4 Results

In what follows, the arithmetic mean of three independent runs for each
experiment are reported. We also recreate the results from Phan-Minh et al.,
(2020), with our own hyperparameters, which partially improved upon the
results reported in the original paper.

###  5.5 Effect of Constrained Training Data

Table 3: Results for different coverage bounds with 100% of the training data (bold as unique best result and italic as unique best deterministic per set) Experiment |  ACC |  NLL |  ECE |  RNK |  minADE1 |  minADE5 |  minFDE1 |  DAC
---|---|---|---|---|---|---|---|---
$\epsilon$=2 |  |  |  |  |  |  |  |
Base |  0.05 |  5.23 |  0.05 |  106.04 |  5.21 |  2.69 |  11.67 |  0.89
Transfer |  0.04 |  5.19 |  0.06 |  82.77 |  5.38 |  2.54 |  11.50 |  0.97
Loss |  0.04 |  5.68 |  0.13 |  124.9 |  5.09 |  2.44 |  10.76 |  0.97
VI |  0.06 |  5.13 |  0.02 |  98.43 |  4.91 |  2.30 |  10.87 |  0.89
GVCL |  0.06 |  4.94 |  0.05 |  74.12 |  4.60 |  2.26 |  10.24 |  0.93
GVCL-Det |  0.05 |  7.53 |  0.36 |  123.93 |  4.87 |  2.57 |  10.43 |  0.93
$\epsilon$=4 |  |  |  |  |  |  |  |
Base |  0.15 |  3.40 |  0.05 |  15.07 |  4.77 |  2.26 |  10.61 |  0.88
Transfer |  0.15 |  3.24 |  0.06 |  11.56 |  4.70 |  2.12 |  10.15 |  0.96
Loss |  0.13 |  3.58 |  0.13 |  16.85 |  4.88 |  2.40 |  10.53 |  0.94
VI |  0.19 |  3.14 |  0.03 |  11.07 |  4.13 |  2.03 |  9.21 |  0.91
GVCL |  0.19 |  2.98 |  0.02 |  9.51 |  3.99 |  1.40 |  8.96 |  0.94
GVCL-Det |  0.15 |  3.60 |  0.18 |  11.85 |  4.55 |  2.26 |  9.93 |  0.91
$\epsilon$=8 |  |  |  |  |  |  |  |
Base |  0.35 |  2.00 |  0.06 |  3.45 |  4.95 |  2.43 |  10.86 |  0.85
Transfer |  0.33 |  1.95 |  0.09 |  3.20 |  4.96 |  2.34 |  10.56 |  0.89
Loss |  0.35 |  1.98 |  0.08 |  3.43 |  4.91 |  2.40 |  10.62 |  0.88
VI |  0.41 |  1.84 |  0.03 |  2.94 |  4.33 |  2.31 |  9.44 |  0.87
GVCL |  0.42 |  1.73 |  0.05 |  2.78 |  4.19 |  2.27 |  9.16 |  0.90
GVCL-Det |  0.37 |  2.10 |  0.19 |  3.08 |  4.68 |  2.33 |  9.78 |  0.89

Tab. 2 evaluates the performance of our algorithm in comparison with the
baseline models and focus on the effect of different amounts of available
observations.

We make the following observations. Our GVCL-CoverNet model outperforms all
baselines in all metrics except the DAC, independent of the number of
available observations. Most notably, the prediction performance of GVCL-
CoverNet as measured by the ACC, minADE1 and minFDE1 is substantially better
compared to the other informed baselines (Transfer- and Loss-CoverNet). We
also observe that GVCL-CoverNet shows a better calibration (NLL and ECE) and
that the NLL and RNK gains significantly over the baselines increase with
lesser amounts of data. This shows that our method is especially useful in
case only limited observations are available.

In case of the DAC, our model is outperformed by the Transfer-CoverNet
baseline. However, we hypothesize that Transfer-CoverNet may overly bias the
DAC as it performs worse in all other metrics. While we can safely assume that
the ground truth trajectory lies in the drivable area, we cannot assume the
same for the respective positive label in trajectory set, due to the
clustering (c.f. 3.2) Thus, our GVCL-CoverNet model may correctly trade-off
the DAC for general performance gains, especially as measured by the ACC. We
also note, that the Loss-CoverNet baseline is not able to consistently
outperform Base-CoverNet even though it improves on the DAC as well.

A caveat is that our GVCL-CoverNet model employs double the parameters due to
the variational formulation. However, we observe that our GVCL-Det-CoverNet
still achieves the best ACC, minADE1 , minFDE1 of all non-variational models.
The RNK and minADE5 show worse performance than Transfer-CoverNet, possible
because substantial calibration is lost when disregarding the already learned
variance information. However, with decreasing amounts of available
observations, the GVCL-Det-CoverNet mode becomes more competitive. In fact, we
can obtain the predictive performance (in terms of minADE1 and minFDE1 ) of
the Base model, with just $50\%$ of the training data. This further
substantiates the usefulness of our method under limited amounts of
observations.

It should be noted, the performance gains can only be partially attributed to
the probabilistic formulation. The non-informed VI-CoverNet model already
improves on the Base-CoverNet model across all metrics, but GVCL-CoverNet
again outperforms the VI-CoverNet model across all metrics. Most notably, the
calibration as measured by the NLL and ECE is increasingly improved the less
observations are available. We see according improvements in the calibration-
senstive RNK and minADE5 metrics and a higher DAC, especially when only 10% of
observations are available.

###  5.6 Effect of Coverage Bound

Tab. 3 investigates the effect of the coverage bounds $\epsilon\in\\{2,4,8\\}$
on the prediction performance, using $100\%$ of the training data. This allows
a closer comparison to the basic CoverNet approach, that was also evaluated
with multiple $\epsilon$ values.

Similar as before, we first observe that our GVCL-CoverNet model still
outperforms all informed baselines for each coverage bound across all metrics
except the DAC. Regarding the ECE, the VI-CoverNet model achieves a better
performance for coverage bounds $\epsilon\in\\{2,8\\}$. However, the ECE is
here not consistent with the NLL, which makes it difficult to draw conclusions
about the calibration of the two models. As in Sec. 5.5, we also observe that
our GVCL-Det-CoverNet model achieves substantial performance gains with the
best ACC, minADE1 , minFDE1 of all non-variational models in all sets of
experiments. However, we see here that the calibration issues of the GVCL-Det-
CoverNet model persist, with the worst NLL and ECE in all sets of experiments.
We again conclude, that it is not optimal to simply discard the variance
information.

Second, we observe that all models achieve their respective best prediction
performance (as measured by minADE1 , minADE5 and minFDE1 ) with a coverage
bound $\epsilon=4$. We assume that a low trajectory set resolution with a
coverage bound $\epsilon=8$ is too coarse to sufficiently cover all possible
trajectories, leading to higher deviation from the actual ground truth, while
a high resolution with coverage bound $\epsilon=2$ poses a too difficult
prediction challenge.

##  6 Conclusion

We propose a probabilistic continual learning approach to integrate prior
knowledge as soft constraint into deep learning models, without assuming any
specific architecture. We introduce an exemplary GVCL-CoverNet application,
that integrates prior drivable area knowledge, using already available map
data, in the multi-modal trajectory prediction for autonomous driving. Our
implementation substantially outperforms the original CoverNet, even when
trained with only half the observations. Furthermore, it outperforms existing
informed CoverNet baselines, with increasing benefits when only a limited
amount of observations is available, even when accounting for advantages of
the probabilistic approach. In result, we clearly demonstrate the benefits of
informing the trajectory prediction through prior knowledge and that a
continual variant is to be preferred over loss- or transfer-based methods.

Future research might explore other probabilistic continual learning methods,
e.g. with subspace approximations, or the integration of multiple prior
knowledge sources, e.g. lane markings and traffic signals in the trajectory
prediction. Furthermore, the optimal method for deriving a deterministic
variant of an probabilistic model should be analysed. Another direction are
more flexible forms of variational densities, such as those allowing for
posterior dependencies in the variational covariance.

#### Acknowledgements

The research leading to these results is funded by the German Federal Ministry
for Economic Affairs and Climate Action within the project “KI Wissen –
Entwicklung von Methoden für die Einbindung von Wissen in maschinelles
Lernen". The authors would like to thank the consortium for the successful
cooperation.

## References

  * Abadi et al., (2016) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D. G., Steiner, B., Tucker, P. A., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., and Zheng, X. (2016).  Tensorflow: A system for large-scale machine learning.  In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA.
  * Bahari et al., (2021) Bahari, M., Nejjar, I., and Alahi, A. (2021).  Injecting Knowledge in Data-driven Vehicle Trajectory Predictors.  Transportation Research Part C: Emerging Technologies.
  * Boulton et al., (2021) Boulton, F. A., Grigore, E. C., and Wolff, E. M. (2021).  Motion Prediction using Trajectory Sets and Self-Driving Domain Knowledge.  arXiv preprint, https://arxiv.org/abs/2006.04767.
  * Caesar et al., (2020) Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020).  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.
  * Cowen-Rivers et al., (2022) Cowen-Rivers, A. I., Lyu, W., Tutunov, R., Wang, Z., Grosnit, A., Griffiths, R. R., Maraval, A. M., Jianye, H., Wang, J., Peters, J., and Ammar, H. B. (2022).  Hebo: Pushing the limits of sample-efficient hyper-parameter optimisation.  Journal of Artificial Intelligence Research, JAIR.
  * Cui et al., (2020) Cui, H., Nguyen, T., Chou, F., Lin, T., Schneider, J., Bradley, D., and Djuric, N. (2020).  Deep kinematic models for kinematically feasible vehicle trajectory predictions.  In Proceedings of the 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France.
  * De Lange et al., (2022) De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G. G., and Tuytelaars, T. (2022).  A continual learning survey: Defying forgetting in classification tasks.  IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Gal and Ghahramani, (2016) Gal, Y. and Ghahramani, Z. (2016).  A theoretically grounded application of dropout in recurrent neural networks.  In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain.
  * He et al., (2015) He, K., Zhang, X., Ren, S., and Sun, J. (2015).  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the 2015 IEEE/CVF International Conference on Computer Vision, ICCV 2015, Santiago, Chile.
  * Huang et al., (2022) Huang, Y., Du, J., Yang, Z., Zhou, Z., Zhang, L., and Chen, H. (2022).  A Survey on Trajectory-Prediction Methods for Autonomous Driving.  IEEE Transactions on Intelligent Vehicles.
  * Kingma and Welling, (2014) Kingma, D. P. and Welling, M. (2014).  Auto-encoding variational bayes.  In Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada.
  * Kirkpatrick et al., (2017) Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. (2017).  Overcoming catastrophic forgetting in neural networks.  Proceedings of the National Academy of Sciences, PNAS.
  * Li et al., (2019) Li, J., Ma, H., and Tomizuka, M. (2019).  Conditional generative neural system for probabilistic trajectory prediction.  In Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China.
  * Liu et al., (2021) Liu, J., Mao, X., Fang, Y., Zhu, D., and Meng, M. Q.-H. (2021).  A Survey on Deep-Learning Approaches for Vehicle Trajectory Prediction in Autonomous Driving.  IEEE Transactions on Intelligent Vehicles.
  * Loo et al., (2021) Loo, N., Swaroop, S., and Turner, R. E. (2021).  Generalized variational continual learning.  In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021, Virtual Event.
  * McAllister et al., (2017) McAllister, R., Gal, Y., Kendall, A., van der Wilk, M., Shah, A., Cipolla, R., and Weller, A. (2017).  Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning.  In Proceedings of the 26th International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia.
  * Mundt et al., (2020) Mundt, M., Hong, Y., Pliushch, I., and Ramesh, V. (2020).  A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning.  arXiv preprint, https://arxiv.org/abs/2009.01797.
  * Nguyen et al., (2018) Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. (2018).  Variational continual learning.  In Proceedings of the 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada.
  * Parisi et al., (2019) Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. (2019).  Continual lifelong learning with neural networks: A review.  Neural Networks.
  * Phan-Minh et al., (2020) Phan-Minh, T., Grigore, E. C., Boulton, F. A., Beijbom, O., and Wolff, E. M. (2020).  Covernet: Multimodal behavior prediction using trajectory sets.  In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.
  * Robbins, (2007) Robbins, H. E. (2007).  A stochastic approximation method.  Annals of Mathematical Statistics.
  * Schwarz et al., (2018) Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. (2018).  Progress & compress: A scalable framework for continual learning.  In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden.
  * von Rueden et al., (2022) von Rueden, L., Houben, S., Cvejoski, K., Bauckhage, C., and Piatkowski, N. (2022).  Informed pre-training on prior knowledge.  arXiv preprint, https://arxiv.org/abs/2205.11433.
  * von Rueden et al., (2021) von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Pfrommer, J., Pick, A., Ramamurthy, R., Walczak, M., Garcke, J., Bauckhage, C., and Schuecker, J. (2021).  Informed Machine Learning – A Taxonomy and Survey of Integrating Knowledge into Learning Systems.  IEEE Transactions on Knowledge and Data Engineering.
  * Wang et al., (2020) Wang, E., Cui, H., Yalamanchi, S., Moorthy, M., and Djuric, N. (2020).  Improving movement predictions of traffic actors in bird’s-eye view models using gans and differentiable trajectory rasterization.  In KDD ’20: Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2020, Virtual Event.
  * Wilson and Izmailov, (2020) Wilson, A. G. and Izmailov, P. (2020).  Bayesian deep learning and a probabilistic perspective of generalization.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS 2020, Virtual Event.
  * Wörmann et al., (2022) Wörmann, J., Bogdoll, D., Bührle, E., Chen, H., Chuo, E. F., Cvejoski, K., van Elst, L., Gleißner, T., Gottschall, P., Griesche, S., Hellert, C., Hesels, C., Houben, S., Joseph, T., Keil, N., Kelsch, J., Königshof, H., Kraft, E., Kreuser, L., Krone, K., Latka, T., Mattern, D., Matthes, S., Munir, M., Nekolla, M., Paschke, A., Pintz, M. A., Qiu, T., Qureishi, F., Rizvi, S. T. R., Reichardt, J., von Rueden, L., Rudolph, S., Sagel, A., Schunk, G., Shen, H., Stapelbroek, H., Stehr, V., Srinivas, G., Tran, A. T., Vivekanandan, A., Wang, Y., Wasserrab, F., Werner, T., Wirth, C., and Zwicklbauer, S. (2022).  Knowledge Augmented Machine Learning with Applications in Autonomous Driving: A Survey.  arXiv preprint, https://arxiv.org/abs/2205.04712.

Supplementary Material:
Informed Priors for Knowledge Integration in Trajectory Prediction

##  Appendix A Hyperparameter Tuning

As reported in the main paper, we individually tune the learning rates and
batch sizes for each model as well as the $\lambda_{\text{{multi}}}$
hyperparameter for the Loss-CoverNet baseline. The tuning is carried out with
100% of the observational training data with coverage bounds
$\epsilon\in\\{2,8\\}$ respectively. The values are selected to minimize the
validation negative log likelihood (NLL) using the heteroscedastic and
evolutionary Bayesian optimizer (HEBO). Here, we summarize the respective
tuning ranges and resulting values for the learning rate and batch size in
Tab. 4. The hyperparameter $\lambda_{{\text{multi}}}$ is tuned in the range
$0.01\dots 1.0$ with $0.01$ selected for both coverage bounds. Note, that we
adopt the results from tuning with $\epsilon=8$ for the experiments with
$\epsilon=4$.

Table 4: Selected values and tuning ranges for batch size and learning rate. Method | Value Batch Size | Range Batch Size | Value Learning Rate | Range Learning Rate
---|---|---|---|---
$\epsilon$=2 |  |  |  |
Base | 5 | 4 $\dots$ 32 | 0.00083 | 0.00001 $\dots$ 0.001
Transfer | 5 | 4 $\dots$ 32 | 0.00083 | 0.00001 $\dots$ 0.001
Loss | 5 | 4 $\dots$ 32 | 0.00083 | 0.00001 $\dots$ 0.001
VI | 12 | 4 $\dots$16 | 0.003 | 0.001 $\dots$ 0.05
GVCL | 12 | 4 $\dots$16 | 0.003 | 0.001 $\dots$ 0.05
$\epsilon$=8 |  |  |  |
Base | 16 | 4 $\dots$ 16 | 0.0008 | 0.00001 $\dots$ 0.001
Transfer | 16 | 4 $\dots$ 16 | 0.0002 | 0.00001 $\dots$ 0.001
Loss | 16 | 4 $\dots$ 16 | 0.0001 | 0.00001 $\dots$ 0.001
VI | 12 | 4 $\dots$ 16 | 0.003 | 0.001 $\dots$ 0.05
GVCL | 12 | 4 $\dots$ 16 | 0.003 | 0.001 $\dots$ 0.05

##  Appendix B Additional Metrics

In addition to the metrics in our main paper results, we evaluated the
minADE10 , minADE15 and HitRate5,2 metrics (Huang et al., 2022). The minADE10
and minADE15 evaluate the quality of the 10 and 15 most likely trajectories in
the prediction respectively. The HitRate5,2 measures if the 5 most likely
trajectories are, roughly said, close enough to the ground truth, given a 2
meter threshold. Similar to the minADE5 metric, these additional metrics
indirectly depend on well calibrated, probabilistic predictions.

We report here a complete summary of the results including these metrics in
Tab. 5 regarding the experiments with constrainted observational training
datasets and in Tab. 6 regarding the experiments with different coverage
bounds $\epsilon\in\\{2,4,8\\}$. Furthermore, we include here in both tables
the standard deviation of the three independent experiment runs. We mainly
omitted this information in our main paper to stay concise. In particular, we
observe that the minADE10 and minADE15 behave similar to the minADE5 metric we
discussed in our main paper. We also see that the HitRate5,2 metric behaves
similar to the discussed minADE1 and minFDE1 metrics. Regarding the standard
deviation, we observe that the Base-CoverNet model exhibits the most
variation. This shows that the prediction quality of the basic CoverNet can
differ substantially.

Table 5: Results for constrained observational training datasets with $\epsilon=4$ (bold as unique best result and italic as unique best deterministic per set) Experiment |  Top1 ACC |  NLL |  ECE |  RNK |  minADE1 |  minADE5 |  minADE10 |  minADE15 |  minFDE1 |  HitRate5,2 |  DAC
---|---|---|---|---|---|---|---|---|---|---|---
100% |  |  |  |  |  |  |  |  |  |  |
Base |  0.15$\pm$0.02 |  3.40$\pm$0.1 |  0.05$\pm$0.02 |  15.07$\pm$1.98 |  4.77$\pm$0.49 |  2.26$\pm$0.10 |  1.74$\pm$0.04 |  1.56$\pm$0.03 |  10,61$\pm$1.16 |  0.19$\pm$0.01 |  0.88$\pm$0.01
Transfer |  0.15$\pm$0.00 |  3.24$\pm$0.01 |  0.06$\pm$0.00 |  11.56$\pm$0.09 |  4.70$\pm$0.04 |  2.12$\pm$0.01 |  1.61$\pm$0.01 |  1.44$\pm$0.01 |  10.15$\pm$0.06 |  0.19$\pm$0.00 |  0.96$\pm$0.01
Loss |  0.13$\pm$0.01 |  3.58$\pm$0.01 |  0.13$\pm$0.02 |  16.85$\pm$1.49 |  4.88$\pm$0.13 |  2.40$\pm$0.05 |  1.74$\pm$0.04 |  1.55$\pm$0.03 |  10.53$\pm$0.27 |  0.18$\pm$0.01 |  0.94$\pm$0.01
VI |  0.19$\pm$0.00 |  3.14$\pm$0.04 |  0.03$\pm$0.00 |  11.07$\pm$0.52 |  4.13$\pm$0.13 |  2.03$\pm$0.09 |  1.60$\pm$0.06 |  1.45$\pm$0.04 |  9.21$\pm$0.35 |  0.21$\pm$0.00 |  0.91$\pm$0.01
GVCL |  0.19$\pm$0.00 |  2.98$\pm$0.02 |  0.02$\pm$0.00 |  9.51$\pm$0.14 |  3.99$\pm$0.05 |  1.94$\pm$0.03 |  1.54$\pm$0.02 |  1.40$\pm$0.01 |  8.96$\pm$0.01 |  0.22$\pm$0.00 |  0.94$\pm$0.01
GVCL Det |  0.15$\pm$0.01 |  3.60$\pm$0.08 |  0.18$\pm$0.02 |  11.85$\pm$0.48 |  4.55$\pm$0.11 |  2.26$\pm$0.05 |  1.72$\pm$0.02 |  1.52$\pm$0.01 |  9.93$\pm$0.25 |  0.19$\pm$0.00 |  0.91$\pm$0.02
50% |  |  |  |  |  |  |  |  |  |  |
Base |  0.12$\pm$0.01 |  3.72$\pm$0.09 |  0.07$\pm$0.01 |  20.88$\pm$1.01 |  5.47$\pm$0.27 |  2.60$\pm$0.12 |  1.92$\pm$0.06 |  1.67$\pm$0.04 |  12.03$\pm$0.69 |  0.16$\pm$0.01 |  0.87$\pm$0.01
Transfer |  0.13$\pm$0.01 |  3.55$\pm$0.02 |  0.07$\pm$0.00 |  15.77$\pm$0.18 |  5.24$\pm$0.11 |  2.33$\pm$0.03 |  1.75$\pm$0.02 |  1.54$\pm$0.01 |  11.14$\pm$0.33 |  0.17$\pm$0.01 |  0.96$\pm$0.01
Loss |  0.11$\pm$0.00 |  3.90$\pm$0.02 |  0.05$\pm$0.02 |  24.83$\pm$0.32 |  5.84$\pm$0.15 |  2.68$\pm$0.04 |  1.98$\pm$0.03 |  1.72$\pm$0.01 |  12.61$\pm$0.25 |  0.15$\pm$0.00 |  0.91$\pm$0.01
VI |  0.16$\pm$0.00 |  3.53$\pm$0.02 |  0.04$\pm$0.01 |  16.49$\pm$0.15 |  4.61$\pm$0.01 |  2.21$\pm$0.02 |  1.73$\pm$0.01 |  1.56$\pm$0.01 |  10.24$\pm$0.03 |  0.19$\pm$0.00 |  0.89$\pm$0.01
GVCL |  0.18$\pm$0.00 |  3.20$\pm$0.02 |  0.02$\pm$0.01 |  12.67$\pm$0.45 |  4.30$\pm$0.03 |  2.08$\pm$0.01 |  1.63$\pm$0.01 |  1.48$\pm$0.01 |  9.57$\pm$0.01 |  0.20$\pm$0.00 |  0.92$\pm$0.00
GVCL Det |  0.14$\pm$0.01 |  3.75$\pm$0.09 |  0.13$\pm$0.01 |  15.67$\pm$0.89 |  4.76$\pm$0.16 |  2.41$\pm$0.09 |  1.84$\pm$0.06 |  1.62$\pm$0.05 |  10.41$\pm$0.42 |  0.18$\pm$0.01 |  0.91$\pm$0.00
10% |  |  |  |  |  |  |  |  |  |  |
Base |  0.09$\pm$0.01 |  4.46$\pm$0.05 |  0.06$\pm$0.01 |  41.37$\pm$1.92 |  6.37$\pm$0.17 |  3.06$\pm$0.08 |  2.26$\pm$0.06 |  1.93$\pm$0.05 |  13.81$\pm$0.33 |  0.13$\pm$0.01 |  0.85$\pm$0.00
Transfer |  0.08$\pm$0.01 |  4.36$\pm$0.08 |  0.12$\pm$0.01 |  32.84$\pm$1.44 |  6.41$\pm$0.14 |  3.14$\pm$0.10 |  2.35$\pm$0.19 |  1.90$\pm$0.04 |  13.41$\pm$0.19 |  0.13$\pm$0.00 |  0.98$\pm$0.00
Loss |  0.08$\pm$0.00 |  4.48$\pm$0.07 |  0.07$\pm$0.01 |  39.51$\pm$1.08 |  6.77$\pm$0.46 |  3.24$\pm$0.14 |  2.38$\pm$0.10 |  2.01$\pm$0.06 |  14.25$\pm$0.19 |  0.11$\pm$0.01 |  0.84$\pm$0.03
VI |  0.10$\pm$0.00 |  4.53$\pm$0.06 |  0.07$\pm$0.01 |  35.61$\pm$1.58 |  6.03$\pm$0.15 |  2.80$\pm$0.08 |  2.06$\pm$0.04 |  1.79$\pm$0.02 |  12.77$\pm$0.22 |  0.14$\pm$0.00 |  0.82$\pm$0.02
GVCL |  0.13$\pm$0.01 |  3.86$\pm$0.05 |  0.03$\pm$0.00 |  27.73$\pm$1.40 |  5.51$\pm$0.12 |  2.55$\pm$0.06 |  1.94$\pm$0.02 |  1.69$\pm$0.01 |  11.86$\pm$0.29 |  0.17$\pm$0.00 |  0.91$\pm$0.01
GVCL Det |  0.11$\pm$0.01 |  4.16$\pm$0.15 |  0.13$\pm$0.03 |  25.66$\pm$2.67 |  5.77$\pm$0.07 |  2.75$\pm$0.05 |  2.06$\pm$0.08 |  1.79$\pm$0.06 |  12.08$\pm$0.09 |  0.16$\pm$0.00 |  0.93$\pm$0.02
Table 6: Results for different coverage bounds with 100% of the observational training dataset (bold as unique best result and italic as unique best deterministic per set) Experiment |  Top1 ACC |  NLL |  ECE |  RNK |  minADE1 |  minADE5 |  minADE10 |  minADE15 |  minFDE1 |  HitRate5,2 |  DAC
---|---|---|---|---|---|---|---|---|---|---|---
$\epsilon$=2 |  |  |  |  |  |  |  |  |  |  |
Base |  0.05$\pm$0,00 |  5.23$\pm$0.03 |  0.05$\pm$0.02 |  106.04$\pm$5.75 |  5.21$\pm$0.21 |  2.69$\pm$0.21 |  1.89$\pm$0.03 |  1.62$\pm$0.01 |  11.67$\pm$0.44 |  0.26$\pm$0.00 |  0,89$\pm$0,00
Transfer |  0.04$\pm$0.00 |  5.19$\pm$0.02 |  0.06$\pm$0.01 |  82.77$\pm$2.06 |  5.38$\pm$0.13 |  2.54$\pm$0.04 |  1.83$\pm$0.03 |  1.52$\pm$0.02 |  11.50$\pm$0.28 |  0.25$\pm$0.00 |  0.97$\pm$0.00
Loss |  0.04$\pm$0.00 |  5.68$\pm$0.07 |  0.13$\pm$0.01 |  124.9$\pm$1.16 |  5.09$\pm$0.04 |  2.44$\pm$0.04 |  1.81$\pm$0.03 |  1.55$\pm$0.02 |  10.76$\pm$0.09 |  0.23$\pm$0.00 |  0.97$\pm$0.01
VI |  0.06$\pm$0.00 |  5.13$\pm$0.01 |  0.02$\pm$0.00 |  98.43$\pm$1.72 |  4.91$\pm$0.1 |  2.30$\pm$0.01 |  1.71$\pm$0.02 |  1.46$\pm$0.02 |  10.87$\pm$0.24 |  0.31$\pm$0.01 |  0.89$\pm$0.00
GVCL |  0.06$\pm$0.00 |  4.94$\pm$0.01 |  0.05$\pm$0.00 |  74.12$\pm$0.95 |  4.60$\pm$0.09 |  2.26$\pm$0.01 |  1.68$\pm$0.01 |  1.42$\pm$0.01 |  10.24$\pm$0.23 |  0.30$\pm$0.01 |  0.93$\pm$0.01
GVCL Det |  0.05$\pm$0.00 |  7.53$\pm$0.59 |  0.36$\pm$0.08 |  123.93$\pm$11.52 |  4.87$\pm$0.09 |  2.57$\pm$0.07 |  1.92$\pm$0.07 |  1.62$\pm$0.06 |  10.43$\pm$0.12 |  0.25$\pm$0.01 |  0.93$\pm$0.00
$\epsilon$=4 |  |  |  |  |  |  |  |  |  |  |
Base |  0.15$\pm$0.02 |  3.40$\pm$0.1 |  0.05$\pm$0.02 |  15.07$\pm$1.98 |  4.77$\pm$0.49 |  2.26$\pm$0.10 |  1.74$\pm$0.04 |  1.56$\pm$0.03 |  10,61$\pm$1.16 |  0.19$\pm$0.01 |  0.88$\pm$0.01
Transfer |  0.15$\pm$0.00 |  3.24$\pm$0.01 |  0.06$\pm$0.00 |  11.56$\pm$0.09 |  4.70$\pm$0.04 |  2.12$\pm$0.01 |  1.61$\pm$0.01 |  1.44$\pm$0.01 |  10.15$\pm$0.06 |  0.19$\pm$0.00 |  0.96$\pm$0.01
Loss |  0.13$\pm$0.01 |  3.58$\pm$0.01 |  0.13$\pm$0.02 |  16.85$\pm$1.49 |  4.88$\pm$0.13 |  2.40$\pm$0.05 |  1.74$\pm$0.04 |  1.55$\pm$0.03 |  10.53$\pm$0.27 |  0.18$\pm$0.01 |  0.94$\pm$0.01
VI |  0.19$\pm$0.00 |  3.14$\pm$0.04 |  0.03$\pm$0.00 |  11.07$\pm$0.52 |  4.13$\pm$0.13 |  2.03$\pm$0.09 |  1.60$\pm$0.06 |  1.45$\pm$0.04 |  9.21$\pm$0.35 |  0.21$\pm$0.00 |  0.91$\pm$0.01
GVCL |  0.19$\pm$0.00 |  2.98$\pm$0.02 |  0.02$\pm$0.00 |  9.51$\pm$0.14 |  3.99$\pm$0.05 |  1.94$\pm$0.03 |  1.54$\pm$0.02 |  1.40$\pm$0.01 |  8.96$\pm$0.1 |  0.22$\pm$0.00 |  0.94$\pm$0.01
GVCL Det |  0.15$\pm$0.01 |  3.60$\pm$0.08 |  0.18$\pm$0.02 |  11.85$\pm$0.48 |  4.55$\pm$0.11 |  2.26$\pm$0.05 |  1.72$\pm$0.02 |  1.52$\pm$0.01 |  9.93$\pm$0.25 |  0.19$\pm$0.00 |  0.91$\pm$0.02
$\epsilon$=8 |  |  |  |  |  |  |  |  |  |  |
Base |  0.35$\pm$0.01 |  2.00$\pm$0.01 |  0.06$\pm$0.01 |  3.45$\pm$0.04 |  4.95$\pm$0.09 |  2.43$\pm$0.04 |  2.17$\pm$0.01 |  2.11$\pm$0.01 |  10.86$\pm$0.24 |  0.08$\pm$0.00 |  0.85$\pm$0.01
Transfer |  0.33$\pm$0.01 |  1.95$\pm$0.01 |  0.09$\pm$0.03 |  3.20$\pm$0.02 |  4.96$\pm$0.1 |  2.34$\pm$0.02 |  2.13$\pm$0.00 |  2.09$\pm$0.00 |  10.56$\pm$0.13 |  0.08$\pm$0.00 |  0.89$\pm$0.02
Loss |  0.35$\pm$0.01 |  1.98$\pm$0.01 |  0.08$\pm$0.02 |  3.43$\pm$0.06 |  4.91$\pm$0.14 |  2.40$\pm$0.05 |  2.16$\pm$0.01 |  2.11$\pm$0.00 |  10.62$\pm$0.31 |  0.08$\pm$0.00 |  0.88$\pm$0.02
VI |  0.41$\pm$0.00 |  1.84$\pm$0.04 |  0.03$\pm$0.01 |  2.94$\pm$0.07 |  4.33$\pm$0.09 |  2.31$\pm$0.01 |  2.13$\pm$0.00 |  2.10$\pm$0.00 |  9.44$\pm$0.22 |  0.08$\pm$0.00 |  0.87$\pm$0.00
GVCL |  0.42$\pm$0.00 |  1.73$\pm$0.01 |  0.05$\pm$0.01 |  2.78$\pm$0.01 |  4.19$\pm$0.03 |  2.27$\pm$0.01 |  2.11$\pm$0.01 |  2.09$\pm$0.00 |  9.16$\pm$0.04 |  0.08$\pm$0.00 |  0.90$\pm$0.01
GVCL Det |  0.37$\pm$0.01 |  2.10$\pm$0.01 |  0.19$\pm$0.02 |  3.08$\pm$0.04 |  4.68$\pm$0.01 |  2.33$\pm$0.01 |  2.12$\pm$0.00 |  2.09$\pm$0.00 |  9.78$\pm$0.10 |  0.08$\pm$0.00 |  0.89$\pm$0.01