# SSL-Lanes: Self-Supervised Learning for Motion Forecasting in Autonomous
Driving

Prarthana Bhattacharyya Chengjie Huang Krzysztof Czarnecki
University of Waterloo, Canada
{p6bhatta, c.huang<EMAIL_ADDRESS>
###### Abstract

Self-supervised learning (SSL) is an emerging technique that has been
successfully employed to train convolutional neural networks (CNNs) and graph
neural networks (GNNs) for more transferable, generalizable, and robust
representation learning. However its potential in motion forecasting for
autonomous driving has rarely been explored. In this study, we report the
first systematic exploration and assessment of incorporating self-supervision
into motion forecasting. We first propose to investigate four novel self-
supervised learning tasks for motion forecasting with theoretical rationale
and quantitative and qualitative comparisons on the challenging large-scale
Argoverse dataset. Secondly, we point out that our auxiliary SSL-based
learning setup not only outperforms forecasting methods which use
transformers, complicated fusion mechanisms and sophisticated online dense
goal candidate optimization algorithms in terms of performance accuracy, but
also has low inference time and architectural complexity. Lastly, we conduct
several experiments to understand why SSL improves motion forecasting. Code is
open-sourced at https://github.com/AutoVision-cloud/SSL-Lanes.

Figure 1: Motion forecasting on Argoverse [8] validation. We show four
challenging scenarios at intersections. The baseline [28] misses all the
predictions. In the first row, our proposed lane masking successfully captures
the right-turn. For the second row, predicting distance to intersection helps
the most in capturing the left turn. In the third row, acceleration at an
intersection is best captured by the model that is made to classify maneuvers
of traffic agents. Finally, in the fourth row, classifying successful final
goal states is the most effective at capturing the left turn. These tasks are
trained with pseudo-labels which are obtained for free from data. Please refer
to Sec. 7.2 for details.

##  1 Introduction

Motion forecasting in a real-world urban environment is an important task for
autonomous robots. It involves predicting the future trajectories of traffic
agents including vehicles and pedestrians. This is absolutely crucial in the
self-driving domain for safe, comfortable and efficient operation. However,
this is a very challenging problem. Difficulties include inherent
stochasticity and multimodality of driving behaviors, and that future motion
can involve complicated maneuvers such as yielding, nudging, lane-changing,
turning and acceleration or deceleration.

The motion prediction task has traditionally been based on kinematic
constraints and road map information with handcrafted rules. These approaches
however fail to capture long-term behavior and interactions with map structure
and other traffic agents in complex scenarios. Tremendous progress has been
made with data-driven methods in motion forecasting [7, 33, 52, 16, 24, 35,
32, 6]. Recent methods use a vector representation for HD maps and agent
trajectories, including approaches like Lane-GCN [28], Lane-RCNN [49], Vector-
Net [12], TNT [52] and Dense-TNT [16]. More recently, the enormous success of
transformers [42] has been leveraged for forecasting in mm-Transformer [32],
Scene transformer [35], Multimodal transformer [20] and Latent Variable
Sequential Transformers [15]. Most of these methods however are extremely
complex in terms of architecture and have low inference speeds, which makes
them unsuitable for real-world settings.

In this work, we extend ideas from self-supervised learning (SSL) to the
motion forecasting task. Self-supervision has seen huge interest in both
natural language processing and computer vision [5] to make use of freely
available data without the need for annotations. It aims to assist the model
to learn more transferable and generalized representation from pseudo-labels
via pretext tasks. Given the recent success of self-supervision with CNNs,
transformers, and GNNs, we are naturally motivated to ask the question: Can
self-supervised learning improve accuracy and generalizability of motion
forecasting, without sacrificing inference speed or architectural simplicity?

Contributions: Our work, SSL-Lanes, presents the first systematic study on how
to incorporate self-supervision in a standard data-driven motion forecasting
model. Our contributions are:

  * •

We demonstrate the effectiveness of incorporating self-supervised learning in
motion forecasting. Since this does not add extra parameters or compute during
inference, SSL-Lanes achieves the best accuracy-simplicity-efficiency trade-
off on the challenging large-scale Argoverse [8] benchmark.

  * •

We propose four self-supervised tasks based on the nature of the motion
forecasting problem. The key idea is to leverage easily accessible map/agent-
level information to define domain-specific pretext tasks that encourage the
standard model to capture more superior and generalizable representations for
forecasting, in comparison to pure supervised learning.

  * •

We further design experiments to explore why forecasting benefits from SSL. We
provide extensive results to hypothesize that SSL-Lanes learns richer features
from the SSL training as compared to a model trained with vanilla supervised
learning.

##  2 Related Work

Motion Forecasting: Traditional methods for motion forecasting primarily use
Kalman filtering [23] with a prior from HD-maps to predict future motion
states [45, 18]. With the huge success of deep learning, recent works use
data-driven approaches for motion forecasting. These methods explore different
architectures involving rasterized images and CNNs [7, 3, 39], vectorized
representations and GNNs [12, 49, 34, 33, 24], point-cloud representations
[46], transformers [35, 32, 15, 20] and sophisticated fusion mechanisms [28],
to generate features that predict final output trajectories. While the focus
of these works is to find more effective ways of feature extraction from HD-
maps and interacting agents, they need huge model capacity, heavy
parameterization, and extensive augmentations or large amounts of data to
converge to a general solution. Other works [52, 6, 40, 50] build on them to
incorporate prior knowledge in the form of predefined candidate trajectories
from sampling or clustering strategies from training data. However the
disadvantage of these methods is that their performance is highly related to
the quality of the trajectory proposals, which becomes an extra dependency.
End-to-end solutions for optimizing end-points of these candidates
trajectories are proposed by Dense-TNT [16] and HOME [14]. Dense-TNT has
state-of-the-art accuracy with a reasonable parameter budget, but its online
dense goal candidate optimization strategy is computationally very expensive,
which is unrealistic for real-time operations like autonomous driving. Lately,
ensembling techniques like MultiPath++ [41] and DCMS [47] have been proposed.
While they have high forecasting performance, a major disadvantage is their
high memory cost for training and heavy computational cost at inference.

Self-supervised Learning: SSL is a rapidly emerging learning framework that
generates additional supervised signals to train deep learning models through
carefully designed pretext tasks. In the image domain, various self-supervised
learning techniques have been developed for learning high-level image
representations, including predicting the relative locations of image patches
[10], jigsaw puzzle [36], image rotation [13], image clustering [4], image
inpainting [38], image colorization [51] and segmentation prediction [37]. In
the domain of graphs and graph neural networks, pretext tasks include graph
partitioning, node clustering, context prediction and graph completion [48,
22, 31, 19]. To the best of our knowledge, this is the first principled
approach that explores motion forecasting for autonomous driving with self-
supervision.

Figure 2: Illustration of the overall SSL-Lanes framework for self-supervision
on motion forecasting through joint training. SSL-Lanes improves upon a
standard-motion forecasting baseline, that consists of an agent encoder, map
encoder, interaction model and a trajectory decoder, trained using a
supervised loss $\mathcal{L}_{sup}$. SSL-Lanes proposes four pretext tasks:
(1) Lane Masking: which recovers feature information from the perturbed lane
graphs. (2) Distance to Intersection: which predicts the distance (in terms of
shortest path length) from all lane nodes to intersection nodes. (3) Maneuver
Classification: predicts the form of a ‘maneuver’ the agent-of-interest
intends to execute (4) Success/Failure Classification: which trains an agent
specialized at achieving end-point goals.

##  3 Problem Formulation

We are given the past motion of $N$ actors. The $i$-th actor is denoted as a
set of its center locations over the past $L$ time-steps. We pre-process it to
represent each trajectory as a sequence of displacements
$\mathcal{P}_{i}=\\{\Delta\bm{p}_{i}^{−L+1},...,\Delta\bm{p}_{i}^{−1},\Delta\bm{p}_{i}^{0}\\}$,
where $\bm{p}_{i}^{l}$ is the 2D displacement from time step $l-1$ to $l$. We
are also given a high-definition (HD) map, which contains lanes and semantic
attributes. Each lane is composed of several consecutive lane nodes, with a
total of $M$ nodes. $\bm{X}\in\mathbb{R}^{M\times F}$ denotes the lane node
feature matrix, where $\bm{x}_{j}=\bm{X}[j,:]^{T}$ is the $F$-dimensional lane
node vector. Following the connections between lane centerlines (i.e.,
predecessor, successor, left neighbour and right neighbour), we represent the
connectivity among the lane nodes with four adjacency matrices
$\\{\bm{A}_{f}\\}_{f\in\\{\text{pre,}\text{suc,}\text{left,}\text{right}\\}}$,
with $\bm{A}_{f}\in\mathbb{R}^{M\times M}$. This implies that if
$\bm{A}_{f,gh}=1$, then node $h$ is an $f$-type neighbor of node $g$. Our goal
is to forecast the future motions of all actors in the scene
$\mathcal{O}^{1:T}_{\text{GT}}=\\{(x^{1}_{i},y^{1}_{i}),...,(x^{T}_{i},y^{T}_{i})|i=1,...,N\\}$,
where $T$ is our prediction horizon.

##  4 Background

In this section, we first briefly introduce a standard data-driven motion
forecasting framework.

Feature Encoding: We first encode the agent and map inputs similar to Lane-GCN
[28]. The agent encoder includes a 1D convolution with a feature pyramid
network, parameterized by $g_{\text{enc}}$, as given by Eq. 1. For map-
encoding, we adopt two Lane-Conv residual blocks, parameterized by
$\bm{\mathrm{\Theta}}=\\{\bm{W}_{0},\bm{W}_{\text{left}},\bm{W}_{\text{right}},\bm{W}_{\text{pre},k},\bm{W}_{\text{suc},k}\\}$,
where $k\in\\{1,2,4,8,16,32\\}$, as given by Eq. 2.

$\small\hat{\bm{p}}_{i}=g_{\text{enc}}(\mathcal{P}_{i})\normalsize$ (1)

$\bm{Y}=\bm{X}\bm{W}_{0}+\sum\limits_{j\in\\{\text{left},\text{right}\\}}{\bm{A}_{j}\bm{X}\bm{W}_{j}}+\sum\limits_{k}{\bm{A}_{\text{pre}}^{k}\bm{X}\bm{W}_{\text{pre},k}+\bm{A}_{\text{suc}}^{k}\bm{X}\bm{W}_{\text{suc},k}}$

(2)

Modeling Interactions: Since the behavior of agents depends on map topology
and social consistency, each encoded agent $i$ subsequently aggregates context
from the surrounding map features and its neighboring agent features, via
spatial attention [43] as given by Eq. 3:

$\small\begin{split}\bm{\tilde{p}}_{i}=\bm{\hat{p}}_{i}\bm{W}_{\text{M2A}}+\sum_{j}{\phi(\text{concat}(\bm{\hat{p}}_{i},\Delta_{i,j},\bm{y}_{j})\bm{W}_{1})\bm{W}_{2}}\\\
\bm{\acute{p}}_{i}=\bm{\tilde{p}}_{i}\bm{W}_{\text{A2A}}+\sum_{j}{\phi(\text{concat}(\bm{\tilde{p}}_{i},\Delta_{i,j},\bm{\tilde{p}}_{j})\bm{W}_{3})\bm{W}_{4}}\end{split}$
(3)

Here, $\bm{y}_{j}$ is the feature of the $j$-th node, $\bm{\hat{p}}_{i}$ is
the feature of the $i$-th agent, $\phi$ the composition of layer normalization
and ReLU, and $\Delta_{ij}=\text{MLP}(\bm{v}_{j}−\bm{v}_{i})$, where $\bm{v}$
denotes the $(x,y)$ 2-D BEV location of the agent or the lane node. The
parameters for map and agent feature aggregation is represented by
$\bm{\Lambda}=\\{\bm{W}_{\text{M2A}},\bm{W}_{1},\bm{W}_{2},\bm{W}_{\text{A2A}},\bm{W}_{3},\bm{W}_{4}\\}$.

Trajectory Prediction: Finally, we decode the future trajectories from the
features $\bm{\acute{p}}_{i}$ corresponding to the agents of interest as given
by:
$\mathcal{O}^{1:T}_{\text{pred}}=\\{g_{\text{dec}}(\bm{\acute{p}}_{i})|i=1,...,N\\}$,
where $g_{\text{dec}}$ is the parameterized trajectory decoder. The parameters
for the motion forecasting model are learned by minimizing the supervised loss
($\mathcal{L}_{\text{sup}}$) calculated between the predicted output and the
ground-truth future trajectories ($\mathcal{O}^{1:T}_{\text{GT}}$), as given
by Eq. 4:

$\small
g_{\text{enc}}^{\star},\bm{\mathrm{\Theta}}^{\star},\bm{\Lambda}^{\star},g_{\text{dec}}^{\star}=\mathop{\text{arg
min}}_{g_{\text{enc}},\bm{\mathrm{\Theta}},\bm{\Lambda},g_{\text{dec}}}\mathcal{L}_{\text{sup}}(\mathcal{O}^{1:T}_{\text{pred}},\mathcal{O}^{1:T}_{\text{GT}})$
(4)

SSL Task | Property Level | Primary Assumption | Type
---|---|---|---
Lane-Masking | Map features | Local map structure | Aux. auto-encoder
Distance to Intersection | Global map structure | Aux. regression
Maneuver Classification |  Map-aware agent features | Agent feature similarity | Aux. classification
Success/Failure Classification | Distance to success state
Table 1: Overview of our proposed self-supervised (SSL) tasks

##  5 SSL-Lanes

The goal of our proposed SSL-Lanes framework is to improve the performance of
the primary motion forecasting baseline by learning simultaneously with
various self-supervised tasks. Fig. 2 shows the pipeline of our proposed
approach, and Tab. 1 summarizes the self-supervised tasks.

###  5.1 Self-Supervision meets Motion Forecasting

Before we discuss designing pretext tasks to generate self-supervisory
signals, we consider a scheme that will allow combined training for self-
supervised pretext tasks and our standard framework.

How to combine motion forecasting and SSL? Self-supervision can be combined
with motion forecasting in various ways. In one scheme we could pre-train the
forecasting encoder with pretext tasks (which can be viewed as an
initialization for the encoder’s parameters) and then fine-tune the pre-
trained encoder with a downstream decoder as given by Eq. 4. In another
scheme, we could choose to freeze the encoder and only train the decoder. In a
third scheme, we could optimize our pretext task and primary task jointly, as
a kind of multi-task learning setup. Inspired by relevant discussions in GNNs,
we choose the third-scheme, i.e., multi-task learning, which is the most
general framework among the three and is also experimentally verified to be
the most effective [48, 22].

Joint Training: Considering our motion forecasting task and a self-supervised
task, the output and the training process can be formulated as:

$\bm{\mathrm{\Psi}}^{\star},\bm{\mathrm{\Omega}}^{\star},\bm{\mathrm{\Theta}}_{\text{ss}}^{\star}=\mathop{\text{arg
min}}_{\bm{\mathrm{\Psi}},\bm{\mathrm{\Omega}},\bm{\mathrm{\Theta}}_{\text{ss}}}\quad\alpha_{1}\mathcal{L}_{\text{sup}}(\bm{\mathrm{\Psi}},\bm{\mathrm{\Omega}})+\alpha_{2}\mathcal{L}_{\text{ss}}(\bm{\mathrm{\Psi}},\bm{\mathrm{\Theta}}_{\text{ss}})$
(5)

where, $\mathcal{L}_{\text{ss}}(\cdot,\cdot)$ is the loss function of the
self-supervised task, $\bm{\mathrm{\Theta}}_{\text{ss}}$ is the corresponding
linear transformation parameter, and $\alpha_{1},\alpha_{2}\in\mathrm{R}_{>0}$
are the weights for the supervised and self-supervised losses. If the pretext
task only focuses on the map encoder, then
$\bm{\mathrm{\Psi}}=\\{\bm{\mathrm{\Theta}}\\}$ and
$\bm{\mathrm{\Omega}}=\\{{g_{\text{enc}},\bm{\Lambda}},g_{\text{dec}}\\}$.
Otherwise,
$\bm{\mathrm{\Psi}}=\\{{g_{\text{enc}},\bm{\mathrm{\Theta}},\bm{\Lambda}}\\}$
and $\bm{\mathrm{\Omega}}=\\{g_{\text{dec}}\\}$. Henceforth, we also define
the following representations. We will represent the primary task encoder as
function $f_{\bm{\mathrm{\Psi}}}$, parameterized by $\bm{\mathrm{\Psi}}$.
Furthermore, given a pretext task, which we will design in the next section,
the pretext decoder $p_{\bm{\mathrm{\Theta}}_{\text{ss}}}$ is a function that
predicts pseudo-labels and is parameterized by
$\bm{\mathrm{\Theta}}_{\text{ss}}$.

Benefit of SSL-Lanes: In Eq. 5, the self-supervised task as a regularization
term throughout network training. It acts as the regularizer learned from
unlabeled data under the minor guidance of human prior (design of pretext
task). Therefore, a properly designed task would introduce data-driven prior
knowledge that improves model generalizability.

###  5.2 Pretext tasks for Motion Forecasting

At the core of our SSL-Lanes approach is defining pretext tasks based upon
self-supervised information from the underlying map structure _and_ the
overall temporal prediction problem itself. Our proposed prediction-specific
self-supervised tasks are summarized in Tab. 1, and assign different pseudo-
labels from unannotated data to solve Eq. 5. Our core approach is simple in
contrast to state-of-the-art that rely on complex encoding architectures [49,
32, 35, 24, 28, 20, 52], ensembling forecasting heads [47, 41], involved final
goal-set optimization algorithms [16, 40] or heavy fusion mechanisms [28], to
improve prediction performance.

Method | $\textrm{minADE}_{1}$ | $\textrm{minFDE}_{1}$ | $\textrm{MR}_{1}$ | $\textrm{minADE}_{6}$ | $\textrm{minFDE}_{6}$ | $\textrm{MR}_{6}$
---|---|---|---|---|---|---
Baseline | 1.42 | 3.18 | 51.35 | 0.73 | 1.12 | 11.07
Lane-Masking | 1.36 | 2.96 | 49.45 |  0.70 | 1.02 | 8.82
Distance to Intersection | 1.38 | 3.02 | 49.53 | 0.71 | 1.04 | 8.93
Maneuver Classification |  1.33 |  2.90 | 49.26 | 0.72 | 1.05 | 9.36
Success/Failure Classification | 1.35 | 2.93 |  48.54 |  0.70 |  1.01 |  8.59
Table 2: Motion forecasting performance on Argoverse validation with our
proposed pretext tasks

####  5.2.1 Lane-Masking

Motivation: The goal of the Lane-Masking pretext task is to encourage the map
encoder $\bm{\mathrm{\Psi}}=\\{\bm{\mathrm{\Theta}}\\}$ to learn local
structure information in addition to the forecasting task that is being
optimized. In this task, we learn by recovering feature information from the
perturbed lane graphs. VectorNet [12] is the only other motion forecasting
work that proposes to randomly mask out the input node features belonging to
either scene context or agent trajectories, and ask the model to reconstruct
the masked features. Their intuition is to encourage the graph networks to
better capture the interactions between agent dynamics and scene context.
However, our motivation differs from VectorNet in two respects: (a) We propose
to use masking to learn local map-structure better, as opposed to learning
interactions between map and the agent. This is an easier optimization task,
and we outperform VectorNet. (b) A lane is made up of several nodes. We
propose to randomly mask out a certain percentage of each lane. This is a much
stronger prior as compared to randomly masking out any node and ensures that
the model pays attention to all parts of the map.

Formulation: Formally, we randomly mask (i.e., set equal to zero) the features
of $m_{a}$ percent of nodes per lane and then ask the self-supervised decoder
to reconstruct these features.

$\small\bm{\mathrm{\Psi}}^{\star},\bm{\mathrm{\Theta}}_{\text{ss}}^{\star}=\mathop{\text{arg
min}}_{\bm{\mathrm{\Psi}},\bm{\mathrm{\Theta}}_{\text{ss}}}\frac{1}{m_{a}}\sum_{i=1}^{m_{a}}\mathcal{L}_{\text{mse}}\Big{(}p_{\bm{\mathrm{\Theta}}_{\text{ss}}}([f_{\bm{\mathrm{\Psi}}}(\tilde{\bm{X}},\bm{A}_{f})]_{\bm{v}_{i}}),\bm{X}_{i}\Big{)}$
(6)

Here, $\tilde{\bm{X}}$ is the node feature matrix corrupted with random
masking, i.e., some rows of $\bm{X}$ corresponding to nodes $\bm{v}_{i}$ are
set to zero. $p_{\bm{\mathrm{\Theta}}_{\text{ss}}}$ is a fully connected
network that maps the representations to the reconstructed features.
$\mathcal{L}_{\text{mse}}$ is the mean squared error (MSE) loss function
penalizing the distance between the reconstructed map features
$p_{\bm{\mathrm{\Theta}}_{\text{ss}}}([f_{\bm{\mathrm{\Psi}}}(\tilde{\bm{X}},\bm{A}_{f})]_{\bm{v}_{i}})$
for node $\bm{v}_{i}$ and its GT features $\bm{X}_{i}$.

Benefit of Lane-Masking: Since Argoverse [8] has imbalanced data with respect
to maneuvers, there are cases when right/left turns, lane-changes,
acceleration/deceleration are missed by the baseline even with multi-modal
predictions. We hypothesize that stronger map-features can help the multi-
modal prediction header to infer that some of the predictions should also be
aligned with map topology. For example, even if an agent is likely to go
straight at an intersection, some of the possible futures should also cover
acceleration/deceleration or right/left turns guided by the local map
structure.

####  5.2.2 Distance to Intersection

Motivation: The Lane-Masking pretext task is from a local structure
perspective based on masking and trying to predict local attributes of the
vectorized HD-map. We further develop the Distance-to-Intersection pretext
task to guide the map-encoder,
$\bm{\mathrm{\Psi}}=\\{\bm{\mathrm{\Theta}}\\}$, to maintain global topology
information by predicting the distance (in terms of shortest path length) from
all lane nodes to intersection nodes. Datasets like Argoverse [8] provide lane
attributes which describe whether a lane node is located within an
intersection. This will force the representations to learn a global
positioning vector of each of the lane nodes.

Formulation: We aim to regress the distances from each lane node to pre-
labeled intersection nodes annotated as part of the dataset. Given $K$ labeled
intersection nodes
$\mathcal{V}_{\text{intersection}}=\\{\bm{v}_{\text{intersection},k}|k=1,...K\\}$,
we first generate reliable pseudo labels using breadth-first search (BFS).
Specifically, BFS calculates the shortest distance $d_{i}\in\mathbb{R}$ for
every lane node $\bm{v}_{i}$ from the given set
$\mathcal{V}_{\text{intersection}}$. The target of this task is to predict the
pseudo-labeled distances using a pretext decoder. If
$p_{\bm{\mathrm{\Theta}}_{\text{ss}}}([f_{\bm{\mathrm{\Psi}}}(\bm{X},\bm{A}_{f})]_{\bm{v}_{i}})$
is the prediction of node $\bm{v}_{i}$, and $\mathcal{L}_{\text{mse}}$ is the
mean-squared error loss function for regression, then the loss formulation for
this SSL pretext task is as follows:

$\bm{\mathrm{\Psi}}^{\star},\bm{\mathrm{\Theta}}_{\text{ss}}^{\star}=\mathop{\text{arg
min}}_{\bm{\mathrm{\Psi}},\bm{\mathrm{\Theta}}_{\text{ss}}}\frac{1}{M}\sum_{i=1}^{M}\mathcal{L}_{\text{mse}}\Big{(}p_{\bm{\mathrm{\Theta}}_{\text{ss}}}([f_{\bm{\mathrm{\Psi}}}(\bm{X},\bm{A}_{f})]_{\bm{v}_{i}}),d_{i}\Big{)}$
(7)

Benefit of Distance to Intersection Task: We hypothesize that since change of
speed, acceleration, primary direction of movement etc. for an agent can
change far more dramatically as an agent approaches or moves away from an
intersection, it is beneficial to explicitly incentivize the model to pick up
the geometric structure near an intersection and compress the space of
possible map-feature encoders, thereby effectively simplifying inference. We
also expect this to improve drivable area compliance nearby an intersection,
which is often a problem for current motion forecasting models.

Method | $\textrm{minADE}_{1}$ | $\textrm{minFDE}_{1}$ | $\textrm{MR}_{1}$ | $\textrm{minADE}_{6}$ | $\textrm{minFDE}_{6}$ | $\textrm{MR}_{6}$ | $\textrm{b-FDE}_{6}$
---|---|---|---|---|---|---|---
NN + Map [8] | 3.65 | 8.12 | 94.0 | 2.08 | 4.02 | 58.0 | -
Jean [33] | 1.74 | 4.24 | 68.56 | 0.98 | 1.42 | 13.08 | 2.12
Lane-GCN [28] | 1.71 | 3.78 | 58.77 | 0.87 | 1.36 | 16.20 | 2.05
LaneRCNN [49] |  1.68 | 3.69 |  56.85 | 0.90 | 1.45 |  12.32 | 2.15
TNT [52] | 1.77 | 3.91 | 59.70 | 0.94 | 1.54 | 13.30 | 2.14
DenseTNT [16] | 1.68 |  3.63 | 58.43 | 0.88 | 1.28 | 12.58 | 1.97
PRIME [40] | 1.91 | 3.82 | 58.67 | 1.22 | 1.55 | 11.50 | 2.09
WIMP [24] | 1.82 | 4.03 | 62.88 | 0.90 | 1.42 | 16.69 | 2.11
TPCN [46] | 1.66 | 3.69 | 58.80 | 0.87 | 1.38 | 15.80 | 1.92
HOME [14] | 1.70 | 3.68 | 57.23 | 0.89 | 1.29 |  8.46 |  1.86
mmTransformer [32] | 1.77 | 4.00 | 61.78 | 0.87 | 1.34 | 15.40 | 2.03
MultiModalTransformer [20] | 1.74 | 3.90 | 60.23 |  0.84 | 1.29 | 14.29 | 1.94
LatentVariableTransformer [15] | - | - | - | 0.89 | 1.41 | 16.00 | -
SceneTransformer [35] | 1.81 | 4.06 | 59.21 |  0.80 |  1.23 | 12.55 |  1.88
| SSL-Lanes (Ours)
---
1.63 |  3.56 |  56.71 |  0.84 |  1.25 | 13.26 | 1.94

Table 3: Comparison of our (best) proposed model and top approaches on the
Argoverse Test. The best results are in bold and underlined, and the second
best is also underlined.

####  5.2.3 Maneuver Classification

Motivation: The Lane-Masking and Distance to Intersection pretext tasks are
both based on extracting feature and topology information from a HD-map.
However, pretext tasks can also be constructed from the overall forecasting
task itself. Thus we propose to obtain free pseudo-labels in the form of a
‘maneuver’ the agent-of-interest intends to execute, and define a set of
‘intentions’ to represent common semantic modes (e.g. change lane, speed up,
slow down, turn-right, turn-left etc.) We call this pretext task Maneuver
Classification, and we expect it to provide prior regularization to
$\bm{\mathrm{\Psi}}=\\{{g_{\text{enc}},\bm{\mathrm{\Theta}},\bm{\Lambda}}\\}$,
based on driving modes.

Formulation: We aim to construct pseudo label to divide agents into different
clusters according to their driving behavior and explore unsupervised
clustering algorithms to acquire the maneuver for each agent. We find that
using naive $k$-Means (on agent end-points) or DBSCAN (on Hausdorff distance
between entire trajectories [1]) leads to noisy clustering. We find that
constrained $k$-means [44] on agent end-points works best to divide trajectory
samples into $C$ clusters equally. We define $C=\\{\text{maintain-
speed},\text{accelerate},\text{decelerate},\text{turn-left},\text{turn-
right},\\\ \text{lane-change}\\}$ and the clustering function as $\rho$. If
$p_{\bm{\mathrm{\Theta}}_{\text{ss}}}(f_{\bm{\mathrm{\Psi}}}(\mathcal{P}_{i},\bm{X},\bm{A}_{f}))$
is the prediction of agent $i$’s intention and
$E_{i}=(x_{i,\text{GT}}^{T},y_{i,\text{GT}}^{T})$ is its ground-truth end-
point, then the learning objective is to classify each agent maneuver into its
corresponding cluster using cross-entropy loss $\mathcal{L}_{ce}$ as:

$\bm{\mathrm{\Psi}}^{\star},\bm{\mathrm{\Theta}}_{\text{ss}}^{\star}=\mathop{\text{arg
min}}_{\bm{\mathrm{\Psi}},\bm{\mathrm{\Theta}}_{\text{ss}}}\mathcal{L}_{\text{ce}}\Big{(}p_{\bm{\mathrm{\Theta}}_{\text{ss}}}(f_{\bm{\mathrm{\Psi}}}(\mathcal{P}_{i},\bm{X},\bm{A}_{f})),\rho(E_{i})\Big{)}$
(8)

Benefit of Maneuver Classification Task: We hypothesize if one can identify
the intention of a driver, the future motion of the vehicle will match that
maneuver, thereby reducing the set of possible end-points for the agent. We
also expect that agents with similar maneuvers will tend to have consistent
semantic representations.

####  5.2.4 Forecasting Success/Failure Classification

Motivation: In contrast to maneuver classification, which provides coarse-
grained prediction of the future, self-supervision mechanisms can also offer a
strong learning signal through goal-reaching tasks which are generated from
the agent’s trajectories. We propose a pretext task called Success/Failure
Classification, which trains an agent specialized at achieving end-point goals
which directly lead to the forecasting-task solution. We expect this to
constrain
$\bm{\mathrm{\Psi}}=\\{{g_{\text{enc}},\bm{\mathrm{\Theta}},\bm{\Lambda}}\\}$
to predict trajectories $\epsilon$ distance away from the correct final end-
point. Conceptually, the more examples of successful goal states we collect,
the better understanding of the target goal of the forecasting task we have.

Formulation: Similar to maneuver classification, we wish to create pseudo-
labels for our data samples. We label trajectory predictions as successful
($c=1$) if the final prediction
$(x_{i,\text{pred}}^{T},y_{i,\text{pred}}^{T})$ is within $\epsilon<2$ of the
final end-point $E_{i}$, and as failure ($c=0$) otherwise. We choose $2$ as
our $\epsilon$ threshold because it is also used for miss-rate calculation
(Sec. 6). In this case, $c\in C=\\{0,1\\}$ is the pseudo-label which belongs
to label set $C$. If the pretext decoder predicts agent $i$’s final-endpoint
as
$p_{\bm{\mathrm{\Theta}}_{\text{ss}}}(f_{\bm{\mathrm{\Psi}}}(\mathcal{P}_{i},\bm{X},\bm{A}_{f}))$,
and given ground-truth end-point $E_{i}$ its success or failure label is
$c_{i}$, then the pretext loss can be formulated as:

$\bm{\mathrm{\Psi}}^{\star},\bm{\mathrm{\Theta}}_{\text{ss}}^{\star}=\mathop{\text{arg
min}}_{\bm{\mathrm{\Psi}},\bm{\mathrm{\Theta}}_{\text{ss}}}\mathcal{L}_{\text{ce}}\Big{(}p_{\bm{\mathrm{\Theta}}_{\text{ss}}}(f_{\bm{\mathrm{\Psi}}}(\mathcal{P}_{i},\bm{X},\bm{A}_{f})),c_{i}\Big{)}$
(9)

Benefit of Success/Failure Classification Task: We hypothesize that this task
will especially provide stronger gains for cases where the final end-point is
not aligned with the general direction of agent movement for majority of
samples given in the dataset, and is thus not well captured by average
displacement based supervised loss functions.

Figure 3: Left: $\text{min-FDE}_{6}$ \- $\text{Miss-Rate}_{6}$ trade-off on
Argoverse Validation. Lower-left is better. We optimize both successfully in
comparison to other popular approaches. Right: We plot min-FDE on Argoverse
Test Set against number of model parameters (in millions) and inference time
(in milli-seconds). We find that there is a trade-off between min-FDE
performance, architectural complexity (as measured by number of parameters)
and computational efficiency (as measured by inference time). Our work
achieves the best trade-off (lower-left).

###  5.3 Learning

As all the modules are differentiable, we can train the model in an end-to-end
way. We use the sum of classification, regression and self-supervised losses
to train the model. Specifically, we use:

$\mathcal{L}=\mathcal{L}_{\text{cls}}+\mathcal{L}_{\text{reg}}+\mathcal{L}_{\text{terminal}}+\mathcal{L}_{\text{ss}}$
(10)

For classification and regression loss design, we adopt the formulation
proposed in [28].
$\mathcal{L}_{\text{terminal}}=\frac{1}{N}\sum_{i=1}^{N}L2\Big{(}(x_{i,\text{pred}}^{T},y_{i,\text{pred}}^{T}),(x_{i,\text{GT}}^{T},y_{i,\text{GT}}^{T})\Big{)}$
is a simple L2 loss that minimizes the distance between predicted final-
endpoints and the ground-truth. This is because $\mathcal{L}_{\text{reg}}$ is
averaged across all time-points $1:T$, and from a practical end user
perspective, minimizing the endpoint loss is much more important than
weighting loss from all time-steps equally. Our proposed pretext tasks
contributes to $\mathcal{L}_{\text{ss}}$. During evaluation, we study each
pretext task separately, and their corresponding loss formulations defined in
Eq. 6, Eq. 7, Eq. 8, Eq. 9 are used for joint training.

##  6 Experiments

Dataset: Argoverse provides a large-scale dataset [8] for the purpose of
training, validating and testing models, where the task is to forecast 3
seconds of future motions, given 2 seconds of past observations. This dataset
has more than 30K real-world driving sequences collected in Miami (MIA) and
Pittsburgh (PIT). Those sequences are further split into train, validation,
and test sets, without any geographical overlap. Each of them has 205,942,
39,472, and 78,143 sequences respectively. In particular, each sequence
contains the positions of all actors in a scene within the past 2 seconds
history, annotated at 10Hz. It also specifies one actor of interest in the
scene, with type ‘agent’, whose future 3 seconds of motion are used for the
evaluation. The train and validation splits additionally provide future
locations of all actors within 3 second horizon labeled at 10Hz, while
annotations for test sequences are withheld from the public and used for the
leaderboard evaluation. HD map information is available for all sequences.

We have two main requirements for the dataset: (a) Scale of Data: Modern
motion forecasting methods and self-supervised learning systems require a
large amount of training data to imitate human maneuvers in complex real-world
scenarios. Thus, the dataset should be large-scale and diverse, such that it
has a wide range of behaviors and trajectory shapes across different
geometries represented in the data. (b) Interesting Scenarios for Forecasting
Evaluation: The dataset should be collected for interesting behaviours by
biasing sampling towards complex observed behaviours (e.g., lane changes,
turns) and road features (e.g., intersections), since we wish to focus on
these cases. We find that on the basis of these requirements, as well as its
popularity in the the motion forecasting community, Argoverse [8] is the best
candidate to showcase our method. Please refer to the supplementary for more
details regarding why we choose to focus on it in comparison to other motion
forecasting benchmarks.

Metrics: ADE is defined as the average displacement error between ground-truth
trajectories and predicted trajectories over all time steps. FDE is defined as
displacement error between ground-truth trajectories and predicted
trajectories at the final time step. We compute $K$ likely trajectories for
each scenario with the ground truth label, where $K=1$ and $K=6$ are used.
Therefore, minADE and minFDE are minimum ADE and FDE over the top $K$
predictions, respectively. Miss rate (MR) is defined as the percentage of the
best-predicted trajectories whose FDE is within a threshold (2 m). Brier-
minFDE is the minFDE plus $(1−p)^{2}$, where $p$ is the corresponding
trajectory probability.

Experimental Details: To normalize the data, we translate and rotate the
coordinate system of each sequence so that the origin is at current position
$t=0$ of ‘agent’ actor and x-axis is aligned with its current direction, i.e.,
orientation from the agent location at $t=−1$ to the agent location at $t=0$
is the positive x axis. We use all actors and lanes whose distance from the
agent is smaller than 100 meters as the input. We train the model on 4 TITAN-X
GPUs using a batch size of 128 with the Adam [25] optimizer with an initial
learning rate of $1×10^{−3}$, which is decayed to $1×10^{−4}$ at 100,000
steps. The training process finishes at 128,000 steps and takes about 10 hours
to complete. For our final test-set submission, we use success/failure
classification as the pretext task, and initialize the map-encoder with the
parameters from a model trained with the lane-masking pretext task. To avoid
overfitting to the general directions that agents move, we augment the data
from each scene for the test-set submission. We rotate all trajectories in a
scene around the scene’s origin by $\gamma$, where $\gamma$ varies from 0 to
360 in 30 intervals. We provide more implementation details in the
supplementary.

##  7 Results

###  7.1 Ablation Studies

Effectiveness of Pretext tasks: We first examine the effect of incorporating
our proposed pretext tasks (Sec. 5) with the standard data-driven motion
forecasting baseline (Sec. 4). While evaluating the importance of our proposed
pretext tasks, we wish to underline that motion prediction for autonomous
driving is a safety-critical task, especially at intersections where most of
our data is collected, and most accidents also happen. We thus posit that in
this situation, even a small error in predicting final locations (FDE) for a
given agent can lead to dangerous potential collisions.

Results in Tab. 2 show that all proposed pretext tasks improve motion
forecasting performance for Argoverse. Specifically, the lane-masking pretext
task improves min-FDE by 8.9% and MR@2m by 20.3%. distance to intersection
improves min-FDE by 7.1% and 19.3%. Maneuver classification improves min-FDE
by 6.3% and MR@2m by 15.4%. We expect that improving the quality of clustering
for maneuvers and thus creating better pseudo-labels will improve this
further. Finally, success/failure classification improves min-FDE by 9.8% and
MR@2m by 22.4%. Moreover, since pretext tasks are not used for inference and
only for training, they also do not add any extra parameters or FLOPs to the
baseline, thereby increasing accuracy but at no cost to computational
efficiency or architectural complexity. We present qualitative results with
the different pretext tasks on several hard cases in Fig. 1.

Figure 4: CKA Feature similarity between feature pairs of baseline and
different pretext tasks. Similarity score is 1 for completely overlapping
features and 0 for completely divergent features.

Similarity in feature space: We analyze the CKA similarity [26] between the
representations learnt by: a model trained with pretext task ‘D2I’ (refers to
distance to intersection task) and baseline; two models trained with different
pretext tasks. In Fig. 4, Base(M2A) refers to $\tilde{\bm{p}}_{i}$, Base(A2A)
refers to $\acute{\bm{p}}_{i}$ (see Eq. 3), ‘Mask’ refers to lane-masking,
‘success/fail’ refers to success or failure classification task and
‘intention’ suggests maneuver classification.

Our main questions are: (a) how much does the pretext task feature differ from
the baseline? (b) do the features from different pretext tasks collapse to the
same feature? First we note that representation learned by D2I does not
collapse to the same representation learned by Mask or Success/Fail or
Intention. Secondly we note that D2I features are quite different from
Base-M2A features $\tilde{\bm{p}}_{i}$ and Base-A2A features
$\acute{\bm{p}}_{i}$, which suggests that task-specific regularization has
indeed resulted in different parameters.

Description | Experimental Setup | Method | $\textrm{minADE}_{6}$ | $\textrm{minFDE}_{6}$ | $\textrm{MR}_{6}$
---|---|---|---|---|---
Training | Validation
Effects of limited training data | 25% of train | All | Baseline | 0.82 | 1.33 | 14.66
Ours |  0.78 |  1.22 |  12.63
Effects of new domain |  100% PIT + 20% MIA | MIA val | Baseline | 0.88 | 1.46 | 17.21
Ours |  0.85 |  1.34 |  14.96
Performance on difficult maneuvers | All |  Turning & lane changing | Baseline | 0.90 | 1.53 | 19.90
Ours |  0.84 |  1.34 |  14.93
Effects of imbalanced data |  2x straight 1x other maneuvers |  Turning & lane changing | Baseline | 0.94 | 1.65 | 21.53
Ours |  0.90 |  1.49 |  17.97
Effects of noisy data | All |  Gaussian noise ($\sigma=0.2$) with $p=0.25$ | Baseline | 1.01 | 1.37 | 15.59
Ours |  0.96 |  1.24 |  11.98
Effects of noisy data | All |  Gaussian noise ($\sigma=0.2$) with $p=0.5$ | Baseline | 1.19 | 1.56 | 20.64
Ours |  1.13 |  1.40 |  15.65

Table 4: Different experimental settings to provide evidence for why SSL-based
training helps motion forecasting

###  7.2 Comparison with State-of-the-Art

Performance: We compare our approach with top entries on Argoverse motion
forecasting leaderboard [8] in Tab. 3. SSL-Lanes improves the metrics for
$K=1$ convincingly and outperforms existing approaches w.r.t. $\text{min-
ADE}_{1}$, $\text{min-FDE}_{1}$ and $\text{MR}_{1}$. We are also strongly
competitive w.r.t. $\text{min-ADE}_{6}$, $\text{min-FDE}_{6}$ and
$\text{MR}_{6}$ against top approaches, with a relatively simple architecture.

Trade-off between min-FDE and Miss-Rate: $\text{min-FDE}_{6}$ and
$\text{MR}_{6}$ are both important for autonomous robots to optimize. Ideally
we wish for both of these metrics to be low. However, there exists a frequent
trade-off between them. We compare this trade-off in Fig. 3(a) with six other
popular motion forecasting models (in terms of citations and GitHub stars),
namely: Lane-GCN [28], Lane-RCNN [28], MultiPath [7], mm-Transformer [32], TNT
[52] and Dense-TNT [16] on the Argoverse validation set. We are on the lowest-
left of meaning we optimize both $\text{min-FDE}_{6}$ and $\text{MR}_{6}$
successfully in comparison to other top models.

Trade-off between accuracy, efficiency and complexity: We are the first to
point out a trade-off that exists for current state-of-the-art motion
forecasting models between forecasting performance, architectural complexity
and inference speed. This is illustrated in Fig. 3(b)-(c). NN+Map [8] (see
Tab. 3) is a simple nearest-neighbor based approach that also uses map-
features, and while it has advantages in terms of fast inference and low model
complexity, the forecasting performance is very low. MultiPath [7] is a very
popular approach that has reasonable accuracy and inference speed but is
parametrically heavy due to it’s use of convolutional kernels. Lane-GCN is a
vector based approach [28] has comparatively fast inference time and high
accuracy, but uses multiple GNN layers which can lead to problems with over-
smoothing for map-encoders [27] and also has a complicated four-stage fusion
mechanism. Lane-RCNN [49] proposes to capture interactions between agents and
map using not just a single vector, but a local interaction graph per agent -
this adds huge number of hyper-parameters to the model and makes it very
complex. Transformer-based models [32, 20, 35, 15] also suffer in this regard.
Scene-transformer for example has 15M parameters and uses heavy augmentation
to prevent overfitting. A light high-performing model is Dense-TNT [16].
However, Dense-TNT’s inference speed on average is 50ms per agent, because it
proposes a time-intensive optimization algorithm to find a dense goal set that
minimizes the expected error of the given set. In contrast to these popular
models, our approach has high accuracy (min-FDE: 1.25m, MR: 13.3%) while also
having low architectural complexity (1.84M parameters) and high inference
speed (3.30 ms). Thus it provides a great balance for application to real-time
safety-critical autonomous robots.

Qualitative Results: We present some multi-modal prediction trajectories on
several hard cases shown in Fig. 1. The yellow trajectory represents the
observed 2s. Red represents ground truth for the next 3s and green represents
the multiple forecasted trajectories for those 3s. In Row 1, the agent turns
right at the intersection. The baseline misses this mode completely, despite
having access to the map. The model trained with lane-masking successfully
predicts this right turn within 2m of the ground-truth end-point. In Row 2,
the agent has a noisy past history and accelerates while turning left at the
intersection. The pretext task distance-to-intersection can correctly capture
this, while the baseline has only one trajectory covering this mode but vastly
overshoots the ground-truth. Interestingly, we note that the success/failure
pretext task is unable to capture this mode. We believe this is due to a
stronger prior imposed by the model during learning. In Row 3, we have an
agent accelerating while going straight at an intersection. We find that the
maneuver classification pretext task is the only model that correctly predicts
trajectories aligned with the ground-truth. In Row 4, we have an agent turning
left at an intersection. Most of the predictions of other models predicts that
the agent will go straight. The success/failure pretext task however picks up
on the left-turn, possibly due to the priors imposed upon it by end-point
conditioning.
Overall, SSL-Lanes can capture left and right turns better, while also being
able to discern acceleration at intersections. Our pretext tasks provide
priors for the model and provides data-regularization for free. We believe
this can improve forecasting through better understanding of map topology,
agent context with respect to the map, and generalization with respect to
imbalance implicitly present in data.

###  7.3 When does SSL help Motion Forecasting?

Hypotheses: We hypothesize that training with SSL pretext tasks probably helps
motion forecasting as following: (a) Topology-based context prediction assumes
feature similarity or smoothness in small neighborhoods of maps. Such a
context-based feature representation can greatly improve prediction
performance, especially when the neighborhoods are small. (b) Clustering and
classification assumes that feature similarity implies target-label similarity
and can group distant nodes with similar features together, leading to better
generalization. (c) Supervised learning with imbalanced datasets sees
significant degradation in performance. Although most of the data samples in
Argoverse are at an intersection, a significantly large number involve driving
straight while maintaining speed. Recent studies [30] have shown that SSL
tends to learn richer features from more frequent classes which also allows it
to generalize to to other classes better.
Experiments: In order to provide evidence for our hypotheses, we propose to
design 6 different training and testing setups as shown in Tab. 4. We use
success/failure classification as the pretext task, and all models are trained
for 50,000 steps. We initialize the map-encoder with the parameters from a
model trained with the lane-masking pretext task.
Our first setting is to train with 25% of the total data available for
training and testing on the full validation set. We expect the SSL-based task
to capture richer features and generalize better than the baseline. Our second
setting assumes that SSL also generalizes to topology from different cities
and trains on 100% of data from Pittsburgh (PIT) but only 20% of data from
Miami (MIA). For evaluation, we only test on data examples taken from the city
of MIA. For our third setting, we assume that SSL learns superior features and
can thus perform better in difficult cases like lane-changes and turning
cases. For evaluation, we only test on data examples which involves these
difficult cases. In our fourth setting, we choose to explicitly train with
data that contains 2$\times$ ‘straight-with-same-speed’ maneuver and 1$\times$
all other maneuvers. We test only on lane-changes and turning cases from
validation. Finally in order to test the effect of noise on motion forecasting
performance, we take two models already trained on full data. We now take the
full validation set, randomly select agent trajectories or map nodes with
probability $p=0.25$ and $p=0.5$, and then add Gaussian noise with zero mean
and $0.2$ variance to their features. We expect this to have the most impact
on forecasting performance as compared to all other settings since this is the
most aggressive form of corruption. But we also expect SSL-based pretext task
training to provide robustness to noise for free due to better generalization
capabilities. Our takeaway from these experiments is that there is strong
evidence SSL-based tasks do provide better generalization capabilities and can
thus prove to be more effective than pure supervised training based
approaches.

##  8 Conclusion

We propose SSL-Lanes to leverage supervisory signals generated from data for
free in the form of pseudo-labels and integrate it with a standard motion
forecasting model. We design four pretext tasks that can take advantage of
map-structure and similarities between agent dynamics to generate these
pseudo-labels, namely: lane masking, distance to intersection prediction,
maneuver classification and success/failure classification. We validate our
proposed approach by achieving competitive results on the challenging large-
scale Argoverse benchmark. The main advantage of SSL-Lanes is that it has high
accuracy combined with low architectural complexity and high inference speed.
We further demonstrate that each proposed SSL pretext task improves upon the
baseline, especially in difficult cases like left/right turns and
acceleration/deceleration. We also provide hypotheses and experiments on why
SSL-Lanes can improve motion forecasting.

Limitations: A limitation of our framework is that it uses the different
losses for our formulation only in a 1:1 ratio without tuning them. We also
use only one pretext task at a time and do not explore the combination of
these different tasks. For our future work, we plan to incorporate meta-
learning [21] to identify an effective combination of pretext tasks and
automatically balance them—we expect that this will lead to more gains in
terms of forecasting performance. Another limitation is that we report
improvements with SSL-pretext tasks in scenarios without specifically
considering multiple heavily interacting agents. In the future we would like
to explore how the interactions between road agents can influence our SSL
losses on the interaction split of the Waymo Open Motion dataset (WOMD) [11].
Finally, we explore generalization in terms of implicit data imbalance only in
comparison to pure supervised training on the same dataset from which training
samples are derived. We would like to study the generalization of our work to
other datasets without re-training.

## Acknowledgements

This research was funded by the Mitacs Accelerate Program and Gatik Inc. This
article solely reflects the opinions and conclusions of its authors.

## References

  * [1] Elmira Amirloo Abolfathi, Mohsen Rohani, Ershad Banijamali, Jun Luo, and Pascal Poupart.  Self-supervised simultaneous multi-step prediction of road dynamics and cost map.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 8494–8503. Computer Vision Foundation / IEEE, 2021.
  * [2] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization.  CoRR, abs/1607.06450, 2016.
  * [3] Mayank Bansal, Alex Krizhevsky, and Abhijit S. Ogale.  Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst.  In Antonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson, editors, Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019.
  * [4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pages 139–156. Springer, 2018.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9630–9640. IEEE, 2021.
  * [6] Sergio Casas, Wenjie Luo, and Raquel Urtasun.  Intentnet: Learning to predict intention from raw sensor data.  In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 947–956. PMLR, 2018.
  * [7] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov.  Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction.  In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 86–99. PMLR, 2019.
  * [8] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays.  Argoverse: 3d tracking and forecasting with rich maps.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8748–8757. Computer Vision Foundation / IEEE, 2019.
  * [9] Nachiket Deo and Mohan M. Trivedi.  Multi-modal trajectory prediction of surrounding vehicles with maneuver based lstms.  In 2018 IEEE Intelligent Vehicles Symposium, IV 2018, Changshu, Suzhou, China, June 26-30, 2018, pages 1179–1184. IEEE, 2018.
  * [10] Carl Doersch, Abhinav Gupta, and Alexei A. Efros.  Unsupervised visual representation learning by context prediction.  In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1422–1430. IEEE Computer Society, 2015.
  * [11] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov.  Large scale interactive motion forecasting for autonomous driving : The waymo open motion dataset.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9690–9699. IEEE, 2021.
  * [12] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid.  Vectornet: Encoding HD maps and agent dynamics from vectorized representation.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 11522–11530. Computer Vision Foundation / IEEE, 2020.
  * [13] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [14] Thomas Gilles, Stefano Sabatini, Dzmitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde.  HOME: heatmap output for future motion estimation.  In 24th IEEE International Intelligent Transportation Systems Conference, ITSC 2021, Indianapolis, IN, USA, September 19-22, 2021, pages 500–507. IEEE, 2021.
  * [15] Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Jim Aldon D’Souza, Samira E. Kahou, Felix Heide, and Christopher Pal.  Latent variable sequential set transformers for joint multi-agent motion prediction.  In 10th International Conference on Learning Representations, ICLR 2022, Conference Track Proceedings, 2022.
  * [16] Junru Gu, Chen Sun, and Hang Zhao.  Densetnt: End-to-end trajectory prediction from dense goal sets.  In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 15283–15292. IEEE, 2021.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016.
  * [18] Adam Houenou, Philippe Bonnifait, Véronique Cherfaoui, and Wen Yao.  Vehicle trajectory prediction based on motion model and maneuver recognition.  In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, Tokyo, Japan, November 3-7, 2013, pages 4363–4369. IEEE, 2013.
  * [19] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec.  Strategies for pre-training graph neural networks.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
  * [20] Zhiyu Huang, Xiaoyu Mo, and Chen Lv.  Multi-modal motion prediction with transformer-based neural network for autonomous driving.  CoRR, abs/2109.06446, 2021.
  * [21] Dasol Hwang, Jinyoung Park, Sunyoung Kwon, Kyung-Min Kim, Jung-Woo Ha, and Hyunwoo J. Kim.  Self-supervised auxiliary learning with meta-paths for heterogeneous graphs.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [22] Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang.  Self-supervised learning on graphs: Deep insights and new direction.  CoRR, abs/2006.10141, 2020.
  * [23] Rudolph Emil Kalman and Others.  A new approach to linear filtering and prediction problems.  Journal of basic Engineering, 82(1):35–45, 1960.
  * [24] Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan.  What-if motion prediction for autonomous driving.  CoRR, abs/2008.10587, 2020.
  * [25] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [26] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton.  Similarity of neural network representations revisited.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR, 2019.
  * [27] Qimai Li, Zhichao Han, and Xiao-Ming Wu.  Deeper insights into graph convolutional networks for semi-supervised learning.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 3538–3545. AAAI Press, 2018.
  * [28] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun.  Learning lane graph representations for motion forecasting.  In ECCV, 2020.
  * [29] Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie.  Feature pyramid networks for object detection.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 936–944. IEEE Computer Society, 2017.
  * [30] Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma.  Self-supervised learning is more robust to dataset imbalance.  CoRR, abs/2110.05025, 2021.
  * [31] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu.  Graph self-supervised learning: A survey.  CoRR, abs/2103.00111, 2021.
  * [32] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou.  Multimodal motion prediction with stacked transformers.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7577–7586. Computer Vision Foundation / IEEE, 2021.
  * [33] Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil.  Multi-head attention for multi-modal joint vehicle motion forecasting.  In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 - August 31, 2020, pages 9638–9644. IEEE, 2020.
  * [34] Abduallah A. Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian G. Claudel.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 14412–14420. Computer Vision Foundation / IEEE, 2020.
  * [35] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Benjamin Sapp, Zhifeng Chen, and Jonathon Shlens.  Scene transformer: A unified multi-task model for behavior prediction and planning.  CoRR, abs/2106.08417, 2021.
  * [36] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, volume 9910 of Lecture Notes in Computer Science, pages 69–84. Springer, 2016.
  * [37] Deepak Pathak, Ross B. Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan.  Learning features by watching objects move.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6024–6033. IEEE Computer Society, 2017.
  * [38] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros.  Context encoders: Feature learning by inpainting.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2536–2544. IEEE Computer Society, 2016.
  * [39] Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom, and Eric M. Wolff.  Covernet: Multimodal behavior prediction using trajectory sets.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 14062–14071. Computer Vision Foundation / IEEE, 2020.
  * [40] Haoran Song, Di Luan, Wenchao Ding, Michael Yu Wang, and Qifeng Chen.  Learning to predict vehicle trajectories with model-based planning.  In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Learning Research, pages 1035–1045. PMLR, 2021\.
  * [41] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, and Benjamin Sapp.  Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction.  CoRR, abs/2111.14973, 2021.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  * [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  * [44] Kiri Wagstaff, Claire Cardie, Seth Rogers, and Stefan Schrödl.  Constrained k-means clustering with background knowledge.  In Carla E. Brodley and Andrea Pohoreckyj Danyluk, editors, Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, pages 577–584. Morgan Kaufmann, 2001.
  * [45] Guotao Xie, Hongbo Gao, Lijun Qian, Bin Huang, Keqiang Li, and Jianqiang Wang.  Vehicle trajectory prediction by integrating physics- and maneuver-based approaches using interactive multiple models.  IEEE Trans. Ind. Electron., 65(7):5999–6008, 2018.
  * [46] Maosheng Ye, Tongyi Cao, and Qifeng Chen.  TPCN: temporal point cloud networks for motion forecasting.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 11318–11327. Computer Vision Foundation / IEEE, 2021.
  * [47] Maosheng Ye, Jiamiao Xu, Xunnong Xu, Tongyi Cao, and Qifeng Chen.  DCMS: motion forecasting with dual consistency and multi-pseudo-target supervision.  CoRR, abs/2204.05859, 2022.
  * [48] Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen.  When does self-supervision help graph convolutional networks?  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10871–10880. PMLR, 2020.
  * [49] Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun.  Lanercnn: Distributed representations for graph-centric motion forecasting.  In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 - Oct. 1, 2021, pages 532–539. IEEE, 2021.
  * [50] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun.  End-to-end interpretable neural motion planner.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8660–8669. Computer Vision Foundation / IEEE, 2019.
  * [51] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful image colorization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science, pages 649–666. Springer, 2016.
  * [52] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov.  TNT: target-driven trajectory prediction.  In Jens Kober, Fabio Ramos, and Claire J. Tomlin, editors, 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pages 895–904. PMLR, 2020.
  * [53] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  CoRR, abs/1904.07850, 2019.

Supplementary Material

##  1 Detailed Network Architecture for Baseline

We provide the detailed network architecture of our baseline in Fig. 5.
For the agent feature extractor, the architecture is similar to [28]. We use
an 1D CNN to process the trajectory input. The output is a temporal feature
map, whose element at $t=0$ is used as the agent feature. The network has
three groups/scales of 1D convolutions. Each group consists of two residual
blocks [17], with the stride of the first block as 2. Feature Pyramid Network
(FPN) [29] fuses the multi-scale features, and applies another residual block
to obtain the output tensor. For all layers, the convolution kernel size is 3
and the number of output channels is 128. Layer normalization [2] and
Rectified Linear Unit (ReLU) are used after each convolution.
The map feature extractor has two LaneConv residual [17] blocks which are the
stack of a LaneConv(1, 2, 4, 8, 16, 32) and a linear layer, as well as a
shortcut. All layers have 128 feature channels. Layer normalization [2] and
ReLU are used after each LaneConv and linear layer.
For the map-aware agent feature (M2A) module, the distance threshold is $12$m.
It is $100$m for the agent-to-agent (A2A) interaction module. The two
interaction modules have two residual blocks, which consist of a stack of an
attention layer and a linear layer, as well as a residual connection. All
layers have 128 output feature channels.
Taking the interaction-aware actor features as input, our trajectory decoder
is a multi-modal prediction header that outputs the final motion forecasting.
For each agent, it predicts $K$ possible future trajectories and confidence
scores. The header has two branches, a regression branch to predict the
trajectory of each mode and a classification branch to predict the confidence
score of each mode.
Key differences with Lane-GCN [28]: Our main difference is we use two Lane-
Conv blocks instead of four as map-feature extractor in order to prevent over-
smoothing in GNNs [27]. We also do not use the four-way fusion proposed by
Lane-GCN and do away with the agent to map (A2M) and the map to map (M2M)
interaction blocks, which saves compute and memory.

Figure 5: Architecture of the baseline model

##  2 Implementation of Pretext Tasks

In this section, we discuss various design decisions for the proposed pretext
tasks.

###  2.1 Lane-Masking

For this pretext task, we mask $m_{a}$ percent of every lane and reconstruct
its features.

Method | $\textrm{m}_{a}$ | $\textrm{minADE}_{6}$ | $\textrm{minFDE}_{6}$ | $\textrm{MR}_{6}$
---|---|---|---|---
Baseline | - | 0.73 | 1.12 | 11.07
Random Masking | 0.4 | 0.71 | 1.03 | 9.11
Lane-Masking | 0.3 | 0.71 | 1.04 | 9.02
Lane-Masking | 0.4 | 0.70 | 1.02 | 8.84
Lane-Masking | 0.5 | 0.71 | 1.05 | 9.31

Table 5: Effect of masking ratio ($m_{a}$) on forecasting performance for
lane-masking task

In Tab. 5, we study the influence of masking ratio on the final forecasting
performance. Random masking refers to masking out $m_{a}$ percent random map
nodes and lane-masking refers to masking out $m_{a}$ percent of lanes in the
map. We finally choose $m_{a}=0.4$ as the most effective parameter for the
lane-masking pretext task, which outperforms random masking. The model infers
missing lane-nodes to produce plausible outputs during reconstruction. We
hypothesize that this reasoning is linked to learning useful representations.

###  2.2 Distance to Intersection

For this pretext task, we explore two different options for framing the
problem of predicting the distance to the nearest intersection node in Tab. 6.
We first explore predicting this distance as a classification task. We group
the lengths into four categories: $d_{ij}=1$, $d_{ij}=2$, $d_{ij}=3$,
$d_{ij}=4$ and $d_{ij}>=5$. We however find that this is harder to optimize
than the regression loss proposed in Eq. 7, which we finally choose as our
loss for the distance to intersection pretext task.

Method | Pretext Loss | $\textrm{minADE}_{6}$ | $\textrm{minFDE}_{6}$ | $\textrm{MR}_{6}$
---|---|---|---|---
Baseline | - | 0.73 | 1.12 | 11.07
Distance to Intersection | Classification | 0.72 | 1.06 | 9.64
Distance to Intersection | Regression | 0.71 | 1.04 | 8.93

Table 6: Effect of pretext loss type on forecasting performance for distance
to intersection task

###  2.3 Maneuver Classification

For this pretext task, we first divide the lateral and longitudinal maneuvers
by choosing a threshold angle of $20$ from the vertical. We next find that
constrained k-means [44] on agent end-points for lateral and longitudinal
maneuvers works best to separate the trajectory samples into different
clusters. This is illustrated in Fig. 6. For differentiating the longitudinal
maneuvers from the lane-change maneuver, we check a combination of the
distance from the lane centerlines for start and stop positions and the
orientations of the nearest centerline for start and stop positions.

Figure 6: Modes of driving from unsupervised clustering of data

###  2.4 Success/Failure Classification

For this pretext task, the primary bottleneck is the fact that the number of
positive examples if far fewer than the number of negative examples. This is
because there are only a few success examples in a $2$m area near the end-
point of a single recorded ground-truth trajectory, while the rest of the
points in the scene can be considered as failure examples. We consider first
setting $\epsilon=3$m, i.e. a wider area for success examples, and then
reducing it to $\epsilon=2$m linearly over the total number of training steps.
We find that this can actually harm the final forecasting performance. We thus
follow [53] to use focal loss to train our auxiliary classification task.

##  3 Qualitative Results

Figure 7: Qualitative results for our proposed SSL-Lanes pretext tasks on the
Argoverse [8] validation set. We outperform the baseline on several difficult
cases at intersections and lane-changes.

We first present some multi-modal prediction trajectories on several hard
cases shown in Fig. 1. SSL-Lanes can capture left and right turns better,
while also being able to discern acceleration at intersections. Our pretext
tasks provide priors for the model and provides data-driven regularization for
free. This can improve forecasting because of better understanding of map
topology, agent context with respect to the map, and also improve
generalization for maneuver imbalance implicitly present in data. We next
provide more visual results of our proposed SSL-Lanes on the Argoverse
validation set in Fig. 7. Generally, these qualitative results demonstrate the
effectiveness of our proposed pretext tasks.

##  4 Discussion: SSL-Lanes vs. State-of-the-Art

We use this section to distinguish our work from methods that we believe have
similar intuition but very different construction, in order to highlight its
novelty and value.

  * •

SSL-Lanes vs. VectorNet [12]: Vector-Net is the only other motion forecasting
work that proposes to randomly mask out the input node features belonging to
either scene context or agent trajectories, and ask the model to reconstruct
the masked features. Their intuition is to encourage the graph networks to
better capture the interactions between agent dynamics and scene context.
However, our motivation differs from VectorNet in two respects: (a) We propose
to use masking to learn local map-structure better, as opposed to learning
interactions between map and the agent. This is an easier optimization task,
and we out-perform VectorNet. (b) A lane is made up of several nodes. We
propose to randomly mask out a certain percentage of each lane. This is a much
stronger prior as compared to randomly masking out any node (which may
correspond to either a moving agent or map) and ensures that the model pays
attention to all parts of the map.

  * •

SSL-Lanes vs. CS-LSTM [9]: CS-LSTM appends the encoder context vector with a
one-hot vector corresponding to the lateral maneuver class and a one-hot
vector corresponding to the longitudinal maneuver class. Subsequently, the
added maneuver context allows the decoder LSTM to generate maneuver specific
probability distributions. This construction however is quite different from
our work because it is not auxiliary in nature - it always outputs and appends
a maneuver to the decoder, even during inference. This we believe is too
strong of a bias for the prediction model, especially given the fact that the
maneuvers are generated using very simple velocity profiles and not from
careful mining of the data. In our conditioning, the maneuvers are mined from
data and the final motion prediction does not depend directly on them. We
believe this design is much more flexible since it allows to generate more
supervisory signals in the form of maneuvers during training, but at the same
time does not require an explicit maneuver to condition the final future
forecast trajectory output during inference.

  * •

SSL-Lanes vs. MultiPath [7]: MultiPath is also not auxiliary in nature: it
factorizes motion uncertainty into intent uncertainty and control uncertainty;
models the uncertainty over a discrete set of intents with a softmax
distribution; and then outputs control uncertainty as a Gaussian distribution
dependent on each waypoint state of the anchor trajectory (corresponding to
the intent). While this construction is highly intuitive and effective by
design, it is very different from our SSL-based construction. Ours is an
auxiliary task which provides supervision during training, and effectively
functions as a regularizer, while being general enough to be used with any
other data-driven motion forecasting model.

##  5 Discussion: Choice of Dataset

We now compare the commonly used motion-forecasting datasets, i.e., nuScenes
[nuScenes], Waymo-Open-Motion-Dataset (WOMD) [11] and Argoverse [8]. We
individually discuss why Argoverse is best positioned to bring out the
benefits of our proposed work.

  * •

Scale of Data: We first compare the dataset size and diversity. We note that
Argoverse is not only larger and more diverse than nuScenes, but also has
greater number of training samples and unique trajectories compared to WOMD.

| nuScenes | WOMD | Argoverse
---|---|---|---
Number of Unique Tracks: | 4.3k | 7.65m | 11.7m
Number of Training Segments: | 1k | 104k | 324k

  * •

Interesting Scenarios for Forecasting Evaluation: We next compare if the
datasets specifically mines for interesting scenarios, which is the area we
want to improve the current baseline. nuScenes was not collected to capture a
wide diversity of complex and interesting driving scenarios. WOMD on the other
hand specifically mines for pairwise interaction scenarios, where the main
objective is to improve forecasting for interacting agents. However, the scope
of our study is to primarily focus on motion at intersections undergoing lane-
changes and turns. We expect the SSL-losses to improve understanding of the
context/environment, trajectory embeddings and address data-imbalance w.r.t.
maneuvers. We leave heavy interaction-based use cases for future work.
Finally, Argoverse mines for interesting motion patterns at intersections,
which involve lane-changes, acceleration/deceleration, and turns. We thus find
this dataset best suited to showcase our proposed method.

  * •

Community focus on Argoverse: We also find that many popular motion
forecasting methods published by the robotics community have also included
evaluations only on the Argoverse dataset including: Lane-GCN, Lane-RCNN,
PRIME, DCMS, TPCN, mm-Transformer, HiVT, Multi-modal Transformer, DSP etc.
This makes it easier for us to position our work with respect to these
approaches.

##  6 Discussion: Potential of this Work

We expect this work to influence real world deployment of SSL forecasting
methods for autonomous driving. Another use case for this work is realistic
behavior generation in traffic simulation. The general construction of the
prediction problem, inspired by [28], enables a generic understanding of how
an object moves in a given environment without memorizing the training data. A
neural network may learn to associate particular areas of a scene with certain
motion patterns. To prevent this, we centre around the agent of interest and
normalize all other trajectory and map coordinates with respect to it. We
predict relative motion as opposed to absolute motion for the future
trajectory. This helps to learn general motion patterns. Reconstructing the
map or predicting distances from map elements are conducted in a frame-of-
reference relative to the agent of interest. This helps in learning general
map connectivity. Following work in pedestrian trajectory prediction, we also
additionally add random rotations to the training trajectories to reduce
directional bias. Furthermore, we provide strong evidence that SSL-based tasks
provide better generalization compared to pure supervised training, thereby
having the ability to effectively reuse the same prediction model across
different scenarios.