# KI-PMF: Knowledge Integrated Plausible Motion Forecasting

Abhishek Vivekanandan1, Ahmed Abouelazm1, Philip Schörner1, J. Marius
Zöllner1,2 1 FZI Research Center for Information Technology, 76131 Karlsruhe,
Germany. {vivekana, abouelazm, schoerner, zoellner} @fzi.de 2 Karlsruhe
Institute of Technology (KIT), Germany.

###### Abstract

Accurately forecasting the motion of traffic actors is crucial for the
deployment of autonomous vehicles at a large scale. Current trajectory
forecasting approaches primarily concentrate on optimizing a loss function
with a specific metric, which can result in predictions that do not adhere to
physical laws or violate external constraints. Our objective is to incorporate
explicit knowledge priors that allow a network to forecast future trajectories
in compliance with both the kinematic constraints of a vehicle and the
geometry of the driving environment. To achieve this, we introduce a non-
parametric pruning layer and attention layers to integrate the defined
knowledge priors. Our proposed method is designed to ensure reachability
guarantees for traffic actors in both complex and dynamic situations. By
conditioning the network to follow physical laws, we can obtain accurate and
safe predictions, essential for maintaining autonomous vehicles’ safety and
efficiency in real-world settings. In summary, this paper presents concepts
that prevent off-road predictions for safe and reliable motion forecasting by
incorporating knowledge priors into the training process.

###### Index Terms:

Safety, Motion Forecasting and Planning, Knowledge Integration

##  I Introduction

In complex driving scenarios, accurate and reliable short-term trajectory
predictions of surrounding actors are essential for maintaining safe and
efficient driving performance. For such traffic actors, their intention is
often multi-modal and exhibits plausibility in their action space. For
modeling such a prediction space, through Deep Neural Networks, this multi-
modality often encompasses uncertainty leading to implausible future states.
In the case of applications where safety is critical, this can lead to fatal
situations due to the network’s inability to learn from prior knowledge, as
shown in [1, 2]. The limitations mentioned above pose significant challenges
to the implementation of these approaches in autonomous driving systems aimed
at ensuring safety.

(a) Fig.1a

(b) Fig.1b

(c) Fig.1c

(d) Fig.1d

Figure 1: The left column represents the two scenes with unrefined
trajectories injected into the scene. The right column represents the impact
of our refinement layer, where we refine the original trajectory set with
priors, allowing us to avoid predicting in the off-road areas.

In this work, we propose a novel multi-stage architecture for integrating
explicit knowledge (we use the term constraints interchangeably to represent
priors throughout this work) into a learning-based trajectory forecasting
model to ensure plausible motion forecasts for the traffic participants. Our
network comprises two stages. The first stage is a deterministic, non-
parametric refinement layer that uses explicit environmental and kinematic
constraints to compute plausible trajectories. We represent the problem of
multi-modal trajectory prediction as a classification problem by constructing
the future reachable states as a set of trajectories following the works by
Phan-Minh et al. [3]. This method of output representation allows us to encode
non-holonomic vehicle constraints into the model, which otherwise would be
difficult. Environmental constraints, derived from HD maps M, are encoded as
polygons to perform a Point-in-Polygon search. This process further refines
the trajectories, preventing any off-road predictions. The advantage of such a
non-parametric search is ensuring that the proposals are interpretable. The
second stage utilizes these refined proposals and combines them with goal
positions [4] to learn a plausible representation space. This method of fusing
lanes and trajectories allows the network to learn reachability-based features
from a scene at a global level for the target actor.

The main contributions of this work are as follows:

  * •

Two staged approach of trajectory generation and prediction with constraint-
based refinement layer which provides feasibility and safety assurance to
exclude off-road trajectories and thereby implausible states.

  * •

Multi-headed attention layers, which focus on learning from interactions
between feasible trajectories and corresponding goal-based reachable lanes.

  * •

Extensive runtime analysis to showcase the real-time capabilities of the
refinement layer.

##  II Related Work

Scene representation and encoding is a functionally important part of the
motion forecasting stack, where the representation is learned from rasterized
inputs through the use of CNNs to extract features as seen in ChauffeurNet
[5]. A sensor-independent feature representation in a spatially discretized
grid is proposed in [6], where a 2D CNN encoder-decoder architecture is used
to predict the environment. Stacking of rasterized images is inducted into the
multimodal prediction architecture through the works from [7] where different
heads are used for regressing trajectories as waypoints and classification
heads for estimating the softmax probabilities of the trajectories.

Rasterization was quickly replaced by vector-based methods such as VectorNet
[8] and LaneGCN [9] due to the prohibitively high compute cost and limitations
arising from the receptive field of CNNs.

Generalization Limitations, the limited generalization and lack of safety
guarantees in trajectory forecasting networks were investigated in [1], where
adversarial scenarios are generated to showcase the off-road predictions of
the networks. State-of-the-art methods such as WIMP [10] and LaneGCN had
higher off-road or physically infeasible trajectories, making them not
generalizable. This empirical evidence prompted us to pivot towards the
explicit incorporation of prior knowledge, with the aim of attaining safety
guarantees in predicted trajectories. Although prior knowledge integration and
plausibility-based checks are done for applications related to 3D object
detectors [11], our work is the first to extend it to motion forecasting
problems.

Vehicular motion constraints for motion forecasting problems have been studied
through the works of [12, 13, 14], where a combination of machine learning-
based approach is overlaid on top of an optimizer to solve for assurance.
While these methods ensure the generation of safe trajectories, they do not
incorporate feasibility guarantees within the learning process but rather as a
post-processing step, which can result in unnecessary computational resource
consumption. The problem of motion forecasting was typically dominated by
regression-based methods such as learning constraints via a kinematic layer
through vehicle dynamics modelling from [15, 16]. While they prevented the
infeasible trajectories based off on turn-rate, they had no upper bounds to
guarantee the off-road predictions from the lack of environmental constraints.
Similarly, Covernet [3] captured the vehicle dynamics into a set of
trajectories to represent the future motion of the actors. They then trained a
network to pick the best trajectory for the observation from the set of
trajectories, making the problem space a classification task. Some limitations
of the discussed approach include the absence of feasibility guarantees for
the generated trajectories, as certain trajectories are derived from dataset
samples, and the reliance on rasterized inputs, which constrains their
applicability in online methods. Although PRIME [17] addresses the integration
of learning and trajectory generation within a single pipeline, it generates
trajectories in Frenet space by varying the terminal lateral distance and
velocity of a Frenet trajectory. This transformation of each actor into a
separate Frenet space, which may lead to challenges in handling interactions
among multiple actors.

In this work, we propose a novel approach that prevents off-road predictions
and guarantees physically valid trajectories through the usage of a non-
parametric refinement layer. The guarantees are derived by including physical
and outer constraints into the trajectory generation and consequently
propagated into the learning components of the network for reliable
predictions.

##  III Methodology

###  III-A Problem formulation

The objective of motion forecasting in autonomous driving deals with
predicting the future possible motions of actors in a given scenario. The
estimated tracks of surrounding actors are assumed to be given through an
existing perception system. Given a target actor $a_{tar}$ for prediction,
trajectory forecasting utilizes not only the past observed trajectories of
$a_{tar}$ but also of other actors in its vicinity $a_{other}$ over a finite
time window $t_{p}$ for accurate prediction of its future trajectory over a
prediction horizon $t_{h}$. A past trajectory $T_{p,i}$ of an actor $a_{i}$ is
represented by a sequence of states, as shown in equation 1, where $s_{i}^{t}$
represents the centerpoint location $(x,y)$ of the actor $i$ at a time $t$ in
a Cartesian coordinate frame. The set of all actors’ past trajectories is
noted as $\Gamma_{p}$.

$T_{p,i}=\left\\{s_{i}^{-t_{p}},s_{i}^{-t_{p}+1},s_{i}^{-t_{p}+2},...,s_{i}^{0}\right\\}$
(1)

Similarly, the future predicted trajectory $T_{h,i}$ of a target actor $a_{i}$
is a sequence of predicted center point coordinates, as seen in equation 2.
Such that the set of all actors’ future trajectories is $\Gamma_{h}$.

$T_{h,i}=\left\\{s_{i}^{1},s_{i}^{2},s_{i}^{3},...,s_{i}^{t_{h}}\right\\}$ (2)

Moreover, we utilize HD map information, as illustrated in Figure 2, where
each lane within a query window is discretized and encoded as a set of
centerline $(x,y)$ points in a coordinate system relative to the prediction
target agent $a_{tar}$. The components involved in the architecture will be
discussed in more detail in the following sections. By incorporating HD map
information, the model can better understand the surrounding environment and
enhance the accuracy of trajectory predictions

Figure 2: The green region defines the local buffer polygon, which is used to
reduce the size of the main polygon area. The blue linear lines represent the
polygons. Here we use a circular buffer (red dotted circle) for concept
explanation. The exteriors of the green polygons are then checked for
collisions to make the computation of the trajectory-in-polygon faster. The x
and y-axis are represented in the city coordinates system.

###  III-B Integration of Domain knowledge: First Stage

The first stage takes the trajectory sets as input along with lane centerlines
(converted to a vector representation from HD map) to produce feasible
trajectories that respect the constraint conditions. To generate an initial
set of trajectories, we rely on a similar bagging algorithm like Covernet [3]
on Argoverse dataset [18] with coverage $\epsilon\simeq 2m$ , where coverage
is the maximum distance between two neighboring trajectories in the set. The
resulting trajectory set $\Gamma_{set}$ is considered to be dynamic. Although
the algorithm is good enough to find a diverse set of trajectories, we still
apply kinematic constraints to fit the curvature assumptions of a trajectory.
This results in a set that guarantees the properties of being smooth and
kinematically feasible.

Coordinate transformation The motion of target actors and HD map are
represented in a city coordinate system [18] whereas the trajectory sets are
in Cartesian coordinates. To define a common relative representation for the
networks to learn from, we use the curvilinear-based relative coordinate
system to project the actors’ motions and trajectories, as can be seen from
lines $1-8$ in algorithm 1.

The Refinement Layer takes in a trajectory set
$\Gamma_{set}=\\{T_{0},T_{1},\ldots,T_{n-1}\\}$ and prunes them based on a
trajectory’s plausibility of existence derived from the HD map. Using prior
knowledge about the lane widths from the Lane centerlines, we can delineate
the boundary regions, allowing us to regress the drivable regions. This
construction results in boundary envelope polygons (blue lines, as can be seen
from Fig. 2) surrounding the outermost centerline, which can then be used to
generate upper and lower sampling bounds for trajectories. A trajectory
$T_{i}$ is plausible and considered valid if and only if $\forall(x,y)\in
T_{i},(x,y)\in Polygon_{b}$. A detailed representation of the refinement layer
workings can be seen from the following algorithm 1.

Algorithm 1 Refinement Layer

0: Trajectory set $\Gamma_{set}$, HD-map $M$

0: $Feasible\,trajectories\,\Gamma_{f}\leftarrow$ Local coordinate system

1: $origin\leftarrow$ Query origin of the target actor $a_{tar}$

2: $tangent\leftarrow$ Query tangent vector of the nearest lane $l$ to
$a_{tar}$

3: $\theta\leftarrow$ Calculate rotation angle from the tangent vector
$tangent$

4: $R\leftarrow$ Rotation matrix with rotation angle $\theta$

5: for each actor $i$ in scene do

6: Calculate displacement offsets $dx,dy$ for waypoints defining the
trajectories

7: $\tilde{dx},\tilde{dy}\leftarrow(dx,dy)\cdot R$ {Rotation into curvilinear
coordinates}

8: end for

9: $Goal_{lanes}\leftarrow$ Query centerline points of nearby lanes to find
reachable goal positions.

10: $Polygon_{b}\leftarrow$ Construct buffer polygon boundaries from a given
lane width from a given lane centerline {In the city coordinate system}

11: $\Gamma_{set}\leftarrow\Gamma_{set}\cdot R$

12: for each trajectory $T_{i}\in\Gamma_{set}$ do

13: for each polygon envelope in $Polygon_{b}$ do

14: if $T_{i}\cap Polygon_{b}\neq\emptyset$ then

15: $\Gamma_{set}\setminus T_{i}$

16: else

17: Continue

18: end if

19: end for

20: end for

21: $\Gamma_{f}\leftarrow\Gamma_{set}\cdot R$ {Rotation to local coordinate
system}

22: return $\Gamma_{f}$, $Goal_{lanes}$

Goal lanes are target lane approximations that guide the networks to learn
“where to go” information within the scene. The “which” information is
provided by picking the best trajectory to fit the “where” embeddings. We use
a simple beam search on the lane centerlines with a radius of $100m$ w.r.t
heading angle of the target actor. This results in possible $Goal_{lanes}$,
which are then passed on to the Lane Encoder part of the network for
extracting features.

Figure 3: High-level overview of the architecture. The Refinement layer takes
two inputs. 1. Lane centerline points in the map coordinate system for a given
query window (in meters) and 2. trajectory sets from Covernet with
$\epsilon=2$ coverage. The refinement layer produces feasible/pruned
trajectories by constructing a lane boundary given the lane-centerline points.
It also produces a list of possible goal positions or reachable lanes where
the target actor could reach for within a given prediction horizon. LaneGCN is
used as a backbone architecture and outputs an embedding representation for
$a_{tar}$ which is then concatenated with the lower parts of the network.

###  III-C Encoder

Given the $Goal_{lanes}$ and feasible trajectories generated by the refinement
layer, we propose learned layers to consume the information and project it
into a high-dimensional vector space. Furthermore, we propose the utilization
of attention layers to learn interactions between trajectories and lanes.
Finally, the encoded feature vector of the trajectories can be extended by the
target actor features to enrich its information before passing the final
trajectories’ representation to a classification head, as shown in Fig. 3.

####  III-C1 Learning Goal lanes representation

$Goal_{lanes}$ is a set of lanes, generated by the refinement layer, that the
target actor can traverse given its current heading direction. A goal lane
$L_{i}\in Goal_{lanes}$ is an ordered set of centerline points on the lane
given the direction of the actor’s motion, such that
$L_{i}\in\mathbb{R}^{n_{i}\times 2}$. Each scene can have several goal lanes
$R$, and each lane $L_{i}$ can have a different number of centerline points
$n_{i}$ as a result of differences in road topology and reachabilities. Thus,
the goal lanes are dynamically padded during pre-processing such that all
lanes in the same scene have the same length $N$. Furthermore, each goal lane
is extended with an additional flag set to zero for locations in the lane that
are padded. Accordingly, the goal lanes are represented as a set
$Goal_{lanes}\in\mathbb{R}^{R\times N\times 3}$. The Lane Encoder utilizes
temporal convolution and bidirectional LSTM to learn representations of the
input $Goal_{lanes}$.

####  III-C2 Learning feasible trajectories’ representation

A feasible trajectory is a reachable state for the $a_{tar}$, adhering to the
road topology and the physical constraints of a vehicle model. The Trajectory
Encoder is designed to consume the set of feasible trajectories,
$\Gamma_{f}\in\mathbb{R}^{D\times t_{h}\times 2}$, to learn a high-dimensional
representation of each trajectory independently using a temporal convolution
and an LSTM.

###  III-D Attention Layer

In order to model the interactions between $Goal_{lanes}$ and feasible
trajectories $\Gamma_{f}$ for, $a_{tar}$ we employ the attention mechanism as
described in [19]. Multi-Head Cross-Attention layer enables the mixing of two
different input sequences, allowing for a more comprehensive understanding of
the relationship between the actor’s probable heading (choose the best lanes)
and the best state to execute (choose the best trajectory). Thus, our Lane-
Trajectory interaction block takes the trajectory and lane embeddings to
perform cross-attention. This is achieved by linear projection of trajectory
embeddings as queries (Q), and lane embeddings as keys (K) and values (V) in
the QKV formulation. The output embeddings from this block are then passed
onto the next Trajectory-Trajectory interaction block based on self-attention.
Multi-Head Self-attention (MHSA) provides the network with an intuitive
understanding of global trajectories, facilitating local routing of
information amongst each other. In other words, the MHSA layer allows
trajectories to communicate with each other, enhancing the overall
understanding of the relationships between trajectories in the set. For
computational efficiency, we use multi-head attention in both these blocks
with eight heads.

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V,$ (3)

####  III-D1 Representations fusion and network output

The feature representation of $a_{tar}$, obtained from the A2A layer of the
LaneGCN, is combined with the feasible trajectories’ representation from the
Trajectory-Trajectory interaction block. This combination informs the
trajectories of interactions between $a_{tar}$ and $a_{others}$ in the scene.
The concatenated representation is then fused using an MLP layer. Finally, the
trajectories’ representation is passed through the classification head to
learn the probability of each trajectory. The classification head is a
sequential module of a linear residual block [20] that maintains the input
dimensionality, followed by a linear layer that modifies the output
dimensionality to be equal in size to the number of feasible trajectories.
Thus, the network output $P$ is a vector with the same size as the number of
feasible trajectories in the scene, such that $P\in\mathbb{R}^{D}$. Each
element in the vector provides an estimate of the occurrence probability of
the corresponding trajectory.

###  III-E Loss Learning and Trajectory Scoring

The classification loss of the network is based on the loss function proposed
in [17, 21]. The proposed loss calculates the cross-entropy loss based on the
distance between the feasible trajectories and the ground truth trajectories
compared to the probabilities estimated by the network. The loss calculation
requires the set of feasible trajectories $\Gamma_{f}\in\mathbb{R}^{D\times
30\times 2}$, the target actor ground truth trajectory
$\Gamma_{G}\in\mathbb{R}^{1\times 30\times 2}$, assuming a prediction horizon
$t_{h}=30$, and the network output $P\in\mathbb{R}^{D}$ over the feasible
trajectories. This loss function motivates the network to assign higher
probabilities to feasible trajectories with a small distance to the ground
truth trajectory. These probabilities are calculated based on the Euclidean
distance between the ground truth trajectory and the feasible trajectories
generated by the refinement layer. Equation 4 illustrates the calculation of
the target probability for the $i^{th}$ feasible trajectory $T_{f,i}$, where
$\tau$ is a hyperparameter denoted as temperature factor

$\psi(T_{f,i},\Gamma_{G})=\frac{exp(-Dist(T_{f,i},\Gamma_{G})/\tau)}{\sum_{j=1}^{k}exp(-Dist(T_{f,j},\Gamma_{G})/\tau)}.$
(4)

$Dist(.)$ is a metric distance that calculates the maximum distance between
two trajectories based on the L2 difference of their $(x,y)$ coordinates per
time step, as demonstrated in equation 5. The predicted network probabilities
$\gamma(\Gamma_{f})$ are calculated by applying softmax activation on the
network output $P$. Finally, the classification loss is calculated as the
cross-entropy loss between predicted probabilities $\gamma(\Gamma_{f})$ and
the target probabilities $\psi(\Gamma_{f},\Gamma_{G})$.

Trajectory selection Additionally, top-k selection of feasible trajectories is
applied in the loss calculation to avoid exhaustive backpropagation of the
loss over all the feasible trajectories. In the loss calculation, the top-k
trajectories are the $k_{top}$ trajectories with the highest ground truth
probabilities, requiring the network to learn a better classification
performance over the most similar trajectories to the ground truth. It should
be clarified that in the network evaluation phase, the top-k trajectories for
computing the evaluation metrics are computed based on the top-k predicted
probabilities by the network and have no access to the ground truth
information at all. Thus, avoiding injecting any bias or ground truth
information into the evaluation process.

$Dist(T_{f,i},T_{f,j})=max\left(\left\|s_{1}^{i}-s_{1}^{j}\right\|_{2}^{2},...,\left\|s_{t_{h}}^{i}-s_{t_{h}}^{j}\right\|_{2}^{2}\right)$
(5)

##  IV Experiments

TABLE I: Quantitative comparison of different metrics on argoverse validation split Models |  LowerBound |  |  |  K=1 |  |  |  K=6 |  |  DAC
---|---|---|---|---|---|---|---|---|---
|  minADE |  minFDE |  minADE |  minFDE |  MR |  minADE |  minFDE |  MR |
Argo-NN+Map (prune) |  Reg |  Reg |  3.38 |  7.62 |  0.86 |  1.68 |  3.19 |  0.52 |  0.94
Argo-NN |  Reg |  Reg |  3.45 |  7.88 |  0.87 |  1.71 |  3.29 |  0.54 |  0.87
Argo-CV |  Reg |  Reg |  3.53 |  7.89 |  0.83 |  - |  - |  - |  0.88
WIMP [10] |  Reg |  Reg |  1.43 |  6.37 |  - |  1.07 |  1.61 |  0.23 |  -
LaneGCN [9] |  Reg |  Reg |  1.71 |  3.78 |  0.67 |  0.90 |  1.77 |  0.26 |  -
Covernet $\epsilon=2$ [3] |  1.02 |  0.68 |  3.46 |  8.70 |  0.78 |  1.85 |  3.83 |  0.44 |  -
LaneGCN (Pre-trained)+Ours |  1.02 |  0.68 |  1.97 |  4.72 |  0.73 |  1.55 |  2.04 |  0.25 |  0.99
LaneGCN (Full)+Ours |  1.02 |  0.68 |  1.82 |  4.23 |  0.69 |  1.52 |  1.94 |  0.22 |  0.99

###  IV-A Dataset

We use the Argoverse-1 motion forecasting dataset [18] which comprises 205942
scenes for training, 39472 for validation. Every scene consists of a target
actor whose past motion is fully observable throughout the scene. A scene also
encompasses other actors, whose motions can be partially or fully observed.
For $a_{other}$ which have missing observations, we apply zero padding to
fully match the shape of the target actor. The training and validation epochs
are run with the aim of predicting the target actor’s future reachable state
given the past observations of itself along with other actors in the scene.
The observation horizon is $2s$ and the prediction horizon is $3s$, both of
them sampled at $10Hz$.

###  IV-B Covernet trajectories

We use the Covernet trajectory set having a $\epsilon=2$ with 2800
trajectories. Although Covernet has a trajectory set with, $\epsilon=3,8$ we
chose to use, $\epsilon=2$ as it has dense coverage enough to justify the
capabilities of our approach. The trajectories are upsampled and translated to
fit the local coordinate system. We also calculate the lower bound achievable
with the calculated trajectory sets, as can be seen from Table I, through
minADE calculation between the ground truth trajectory and the closest
trajectory from the set. This is the theoretical lower limit achievable by
training.

###  IV-C Metrics

We evaluate our approach using standard metrics employed in prior studies,
namely the minimum Average Displacement Error (minADE), minimum Final
Displacement Error (minFDE), and Miss Rate (MR) for both single-modal (k = 1)
and multi-modal (k = 6) predictions. Along with these standard metrics, we
provide evaluations for drivable area compliance (DAC) [18]. For a model that
produces $a$ trajectories out of which $b$ trajectories violate road boundary
constraints, the DAC for the model results in $(a-b)/a$. A higher DAC
indicates higher compliance with the drivable area. The minADE metric measures
the minimum average Euclidean error between the predicted trajectory and the
ground-truth trajectory of the target vehicle. Similarly, minFDE measures the
minimum Euclidean error between the predicted endpoint and the ground-truth
endpoint. The ratio of sequences in which none of the predicted endpoints
exceeds $2$ meters from the ground-truth endpoint is called the MR.

###  IV-D Training details

The complete network is trained for $38$ epochs with Adam optimizer [22] on a
NVIDIA A100 GPU. The optimizer has a learning rate of $10^{-3}$ for the first
30 epochs and changes to $10^{-4}$ for the next 8 epochs; with a batch size of
$32$. All layers in the architecture have a dimension of $128$. The number of
heads in the attention layer is $8$.The temperature factor $\tau$ and
$k_{top}$ are set to $1$ and $6$ in the loss calculation.

###  IV-E Computational performance

Although the refinement layer does not have any learnable parameters, it
requires computation time to perform the refinement, which is crucial for the
real-time deployment of the network. In our refinement layer we use the
trajectory-intersects-polygon operation (similar to point-in-polygon), on
average per scene, we were able to perform a check in $0.3-0.5ms$ for $4$
polygon envelops on 12 core CPU. The intersection test could be parallelized
easily, allowing for a significantly faster computation on a GPU. On average,
there were 948 trajectories available after the refinement layer, with four
goal positions constructed per $a_{tar}$ per scene. The Fig. 4 showcases the
theoretical limits for a point-in-polygon operation, which we performed using
different algorithms on a CPU for a total of 1 million randomly sampled
points.

Figure 4: Performance of various methods to employ Point-in-Polygon methods.
We use the ray tracing method in our refinement layer.

##  V Results and Discussion

The quantitative results of our work are compared with similar methods on the
Argoverse validation split dataset, as presented in Table I. To ensure
fairness and consistency in the comparison, we only report the results for
methods that have provided values for all $k=1$, $k=6$, and MR. Our main focus
is on preventing off-road predictions, as evidenced by the high Driving Area
Compliance (DAC) score of 0.99. This indicates that our approach effectively
eliminates almost all infeasible reachable states from the prediction space
used by the network. By comparing our results with the Covernet and LaneGCN,
our model performed relatively well on both the minADE and minFDE aspects for
$k=6$ category, given the lower bound for our model was around 1.02. The MR of
0.22 for our LaneGCN (Full)+Ours was the best result when compared with the
State-of-the-art, as this variant was trained from scratch without using
pretrained weights for the backbone, unlike LaneGCN (pre-trained)+Ours. This
suggests that our model trained from scratch focuses on learning more concrete
representations of plausible states when presented with future reachable
states and goal-oriented lanes, without compromising on the quality of the
predictions.

##  VI Conclusion and future work

In this paper, we introduce a novel framework for multi-modal motion
forecasting that ensures the physical feasibility of trajectories by
incorporating road topology preventing off-road predictions. Our approach
follows a two-step modular process. Firstly, a deterministic refinement layer
is employed to generate physically reachable trajectories and goal positions.
Secondly, a learnable layer captures global interactions among maps, reachable
states, and feasible trajectories, providing comparative results to
regression-based approaches. Going forward, we intend to explore methods to
reduce the computational load of our proposed network by investigating joint
trajectory forecasting involving all actors in the scene, rather than solely
the target actor.

## ACKNOWLEDGMENT

The research leading to these results was funded by the German Federal
Ministry for Economic Affairs and Climate Action and was partially conducted
in the projects “KI Wissen” and “SafeADArchitect”. Responsibility for the
information and views set out in this publication lies entirely with the
authors.

## References

  * [1] Mohammadhossein Bahari et al.  “Vehicle trajectory prediction works, but not everywhere”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17123–17133
  * [2] Micahel Woon  “Physics 101: Violating Universal Laws”  RESEARCH ̵REPORT: https://www.retrospectav.com/products/cruise-unacceptably-risky-part-1
  * [3] Tung Phan-Minh et al.  “Covernet: Multimodal behavior prediction using trajectory sets”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 14074–14083
  * [4] Junru Gu, Chen Sun and Hang Zhao  “Densetnt: End-to-end trajectory prediction from dense goal sets”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15303–15312
  * [5] Mayank Bansal, Alex Krizhevsky and Abhijit Ogale  “Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst”  In _preprint arXiv:1812.03079_ , 2018
  * [6] Philip Schörner et al.  “Grid-based Micro Traffic Prediction using Fully Convolutional Networks”  In _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , 2019, pp. 4540–4547  IEEE
  * [7] Henggang Cui et al.  “Multimodal trajectory predictions for autonomous driving using deep convolutional networks”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 2090–2096  IEEE
  * [8] Jiyang Gao et al.  “Vectornet: Encoding hd maps and agent dynamics from vectorized representation”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11525–11533
  * [9] Ming Liang et al.  “Learning Lane Graph Representations for Motion Forecasting”  In _Computer Vision – ECCV 2020_ Cham: Springer International Publishing, 2020, pp. 541–556
  * [10] Siddhesh Khandelwal et al.  “What-if motion prediction for autonomous driving”  In _arXiv preprint arXiv:2008.10587_ , 2020
  * [11] Abhishek Vivekanandan, Niels Maier and J. Marius Zöllner  “Plausibility Verification for 3D Object Detectors Using Energy-Based Optimization”  In _Computer Vision – ECCV 2022 Workshops_ Springer Nature Switzerland, 2023, pp. 602–616
  * [12] Philip Schörner, Mark Timon Hüneberg and J Marius Zöllner  “Optimization of Sampling-Based Motion Planning in Dynamic Environments Using Neural Networks”  In _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020, pp. 2110–2117  IEEE
  * [13] Matt Vitelli et al.  “SafetyNet: Safe Planning for Real-World Self-Driving Vehicles Using Machine-Learned Policies”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 897–904  DOI: 10.1109/ICRA46639.2022.9811576
  * [14] Meixin Zhu et al.  “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving”  In _Transportation Research Part C: Emerging Technologies_ 117 Elsevier, 2020, pp. 102662
  * [15] Henggang Cui et al.  “Deep Kinematic Models for Kinematically Feasible Vehicle Trajectory Predictions”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 10563–10569  DOI: 10.1109/ICRA40945.2020.9197560
  * [16] Harshayu Girase et al.  “Physically feasible vehicle trajectory prediction”  In _arXiv preprint arXiv:2104.14679_ , 2021
  * [17] Haoran Song et al.  “Learning to predict vehicle trajectories with model-based planning”  In _Conference on Robot Learning_ , 2022, pp. 1035–1045  PMLR
  * [18] Ming-Fang Chang et al.  “Argoverse: 3d tracking and forecasting with rich maps”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 8748–8757
  * [19] Ashish Vaswani et al.  “Attention is All you Need”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017  URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [20] Kaiming He et al.  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [21] Hang Zhao et al.  “Tnt: Target-driven trajectory prediction”  In _Conference on Robot Learning_ , 2021, pp. 895–904  PMLR
  * [22] Diederik P. Kingma and Jimmy Ba  “Adam: A Method for Stochastic Optimization”  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015  URL: http://arxiv.org/abs/1412.6980