# STF: Spatial Temporal Fusion for Trajectory Prediction

Pengqian Han,1 Partha Roop, 1 Jiamou Liu, 1 Tianzhe Bao, 2 Yifei Wang, 1
1 The University of Auckland, Auckland, New Zealand
2 University of Health and Rehabilitation Sciences, Qingdao, China
<EMAIL_ADDRESS>{p.roop<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Trajectory prediction is a challenging task that aims to predict the future
trajectory of vehicles or pedestrians over a short time horizon based on their
historical positions. The main reason is that the trajectory is a kind of
complex data, including spatial and temporal information, which is crucial for
accurate prediction. Intuitively, the more information the model can capture,
the more precise the future trajectory can be predicted. However, previous
works based on deep learning methods processed spatial and temporal
information separately, leading to inadequate spatial information capture,
which means they failed to capture the complete spatial information.
Therefore, it is of significance to capture information more fully and
effectively on vehicle interactions. In this study, we introduced an
integrated 3D graph that incorporates both spatial and temporal edges. Based
on this, we proposed the integrated 3D graph, which considers the cross-time
interaction information. In specific, we design a Spatial-Temporal Fusion
(STF) model including Multi-layer perceptions (MLP) and Graph Attention (GAT)
to capture the spatial and temporal information historical trajectories
simultaneously on the 3D graph. Our experiment on the ApolloScape Trajectory
Datasets shows that the proposed STF outperforms several baseline methods,
especially on the long-time-horizon trajectory prediction. Our code is
available at https://github.com/pengqianhan/STF-Spatial-Temporal-Fusion-for-
Trajectory-Prediction

###### Index Terms:

Graph Neural Network, Trajectory Prediction, Spatial-Temporal Data Mining

##  I Introduction

The trajectory prediction is to accurately predict the future path of an
object based on its historical position. Trajectory data is a type of spatio-
temporal data that contains both spatial and temporal information. Spatial
information pertains to the interaction between objects due to their relative
position, movement tendency, or relative velocity. Besides physical
interaction, there is social interaction between vehicles. Because traffic can
not consider all the driving behavior and the drivers do not always follow the
traffic rules, sometimes they prefer to reach their destination in the
shortest time[1]. Temporal information refers to the time at which the object
is at a specific location or moving from one location to another.

The prediction of the trajectories of surrounding vehicles and pedestrians is
vital for an autonomous vehicle, enabling it to make informed decisions to
avoid collisions. On-board and off-board sensors are used to detect other
vehicles, lanes, and crosswalks and generate large amounts of data that can be
used to train high-performance deep learning models. These models can predict
feasible and efficient trajectories, improving safety and traffic conditions.
Machine learning methods such as Hidden Markov Models (HMMs), Support Vector
Machines (SVMs), and Dynamic Bayesian Networks (DBNs) have been used to
address this problem. However, these approaches are limited to short time
horizons, like 2 seconds, and their prediction performance is unsatisfactory.
Deep learning is a subfield of machine learning that utilizes multiple layers
in its neural networks and has demonstrated significant efficacy with the
increase in data volume[2]. It enables computers to recognize objects from
images or translate languages. When applied to trajectory prediction, deep
learning methods have proven to outperform conventional machine learning
methods. In addition, some work based on deep learning methods like [3, 4, 5,
6, 7] have demonstrated exceptional performance in long-term trajectory
prediction tasks. However, these works process the spatial and temporal
separately. As depicted in Fig. 2, certain studies, such as CS LSTM[8], only
consider spatial information at the final time step. On the other hand,
GRIP++[9] employs graph neural networks (GNNs) to capture spatial information
and subsequently utilize customized convolutional neural networks (CNNs) to
capture temporal information. Additionally, Social LSTM [4] and Starnet [10]
adopt pooling methods to capture the spatial characteristics of historical
trajectories before employing LSTM [11] for capturing temporal information.
All of these methods missed the interaction information across the time frames
during history time. The vehicles on the road interact with each other,
occurring within the same time step and across multiple time steps. Previous
research has focused on the interaction between agents. However, they have
often treated spatial and temporal information in isolation, resulting in a
lack of consideration for interaction related to agents at different time
intervals. In this work, we define an integrated 3D graph with spatial edges
between the nodes at the same time stamp and temporal edges across the time
stamp illustrated in Fig. 2. After defining the 3D graph, all kinds of vehicle
interactions are considered, enhancing the model to capture more information
from the historical trajectory. Finally, we propose a Spatial-Temporal Fusion
(STF) model to predict the future trajectory. Our approach effectively
captures more historical trajectory data information than previous models. We
conducted several experiments that demonstrated significantly improved
performance over existing methods. The code is available on
https://github.com/. The main contribution of this work can be summarized as
follows:

  * •

This study introduces an integrated 3D graph that incorporates both spatial
and temporal edges. The spatial edges represent the spatial interaction
between vehicles within the same time frames, while the temporal edges
indicate cross-time frame interactions, referred to as spatial-temporal
interactions.

  * •

A high-performance trajectory prediction model called STF is proposed in this
work. This model effectively captures a wider range of information from
historical trajectories, including not only spatial interactions but also
spatial-temporal interactions. Additionally, it successfully incorporates
temporal information into its predictions.

  * •

Several experiments are conducted to evaluate the performance of the proposed
approach, demonstrating superior results compared to previous research papers.

Figure 1: 2D graph in previous works

Figure 2: Integrated 3D graph in S-T Fusion

##  II Related works

The methods in previous works are very similar for capturing the temporal
information, most of them choosing the RNNs-based models, including LSTM and
GRU. However, they apply various algorithms to capture the spatial information
of trajectory. These deep learning methods for trajectory prediction can be
classified into three categories.

###  II-A Pooling and CNNs based method

To capture information regarding pedestrian interaction, Social LSTM [4]
implemented a grid-based social pooling technique between LSTM cells. Although
social pooling addresses the issue of varied numbers of neighboring
pedestrians, it does not consider different weights assigned to neighbor
agents. TraPHic [12] also utilizes social pooling to capture interaction
information. The operation regions in TraPHic’s social pooling consist of a
Horizon map and a Neighbor Map, which enable TraPHic to assign different
weights to different agents via social pooling in two maps. It improves the
Social LSTM but still misses the cross-time interaction. In CS LSTM [8],
shared weights LSTM encoder output is used to make a social tensor, which is
put through Convolutional Social Pooling. The Convolutional Social Pooling
captures interaction at the last time of the encoder, but it ignores long-
range interactions. Social GAN [3] designs a Pooling Module to capture the
interaction of all pedestrians in a scene, whereas Social Pooling [4] can only
handle people in an $m\times n$ grid. The relative positions between the
target pedestrian and others are fed into an MLP whose output is concatenated
with the target’s hidden states. The resulting tensor is then processed by
another MLP following a max pooling layer.

###  II-B GNNs based method

The Social-STGCNN model [5] utilizes Graph Convolutional Networks (GCNs) to
capture spatial information and model pedestrian interactions based on a graph
defined at each historical frame. Similarly, GRIP++ [9] defines graphs that
are then processed by GCNs and 2D temporal convolution layers to capture
spatial and temporal information from observed tracks. GSTCN [13] defines a
complete graph with weighted edges based on reciprocal distances between nodes
to model vehicle interactions. ReCoG [14] uses a directed graph where edges
connect vehicles and the target if their distance is below 20 meters. RSBG
[15] trains a Recursive Social Behavior Generator to output weighted edges
representing latent social relationships, which are then fed into GCN layers.
SGCN [16] addresses the issue of superfluous edges by first defining a dense
undirected graph and then converting it into a sparse directed graph to model
pedestrian interactions. The Social-BiGAT model [17] employs GAT and a latent
encoder to process hidden states and generate latent noise, enhancing model
multimodality. All the graph-based deep learning methods miss the traffic
participants’ interaction cross time because they define the graph at every
frame, not the whole historical time.

###  II-C Attention based method

LaneGCN [18] utilizes attention mechanisms to learn interactions among lanes,
actors, and actors themselves. VectorNet [19] employs a hierarchical graph,
with subgraphs representing agent trajectories and road features, which are
processed using an attention mechanism. HiVT [20] adopts a hierarchical
approach by capturing local interactions based on relative positions using an
attention mechanism. It then applies a temporal transformer to generate
spatial and temporal features of the central agent. Another attention
mechanism encodes lane information into keys and values using an MLP and
calculates pairwise global interaction. These papers treated temporal
information and social interaction separately. The ignorance of cross-time
interaction leads to reduced model performance.

##  III PROBLEM FORMULATION

Let $N$ denote the number of traffic agents, $T_{his}$ represent the history
time, and $T_{pred}$ indicate the prediction time. The input history
trajectories set of agents is denoted by
$\mathbf{X}=[\mathbf{x}^{1},\mathbf{x}^{2},...,\mathbf{x}^{T_{his}}]$, where
$\mathbf{x}^{t}$ represents the coordinates of N agents at time $t$, i.e.,
$\mathbf{x}^{t}=[x_{0}^{t},y_{0}^{t},x_{1}^{t},y_{1}^{t},...,x_{N-1}^{t},y_{N-1}^{t}]$.

The prediction trajectories are defined as $\mathbf{Y}$, which includes future
positions from time $(T_{his}+1)$ to $(T_{his}+T_{pred})$, i.e.,
$\mathbf{Y}=[\mathbf{y}^{T_{his}+1},\mathbf{y}^{T_{his}+2},...,\mathbf{y}^{T_{his}+T_{pred}}]$.

##  IV Spatial temporal fusion model

###  IV-A Scheme of model

Fig. 3 illustrates the architecture of the proposed model, which utilizes the
historical trajectory of traffic agents, including cars, bikes, and
pedestrians, as input. The S-T fusion block extracts both spatial and temporal
information simultaneously from this input. Meanwhile, the ST-GCN block
separately captures spatial and temporal information. These two blocks’
outputs are concatenated before being fed into the Seq2Seq network [21] for
predicting future trajectories within a time period of $T_{pred}$.

Figure 3: Scheme of ST fusion model. The 2D graph is defined on all historical
time stamps, while the 3D graph is defined based on the entire historical time
stamps, including spatial edges and temporal edges

###  IV-B Integrated 3D graph definition

In order to process the spatial and temporal information simultaneously, this
work creates a big graph where each node represents a traffic agent, such as
bikes, cars, and pedestrians. An edge is created between two nodes at each
time stamp if their distance is shorter than the threshold $D_{close}$. These
edges are considered spatial edges. Additionally, edges between nodes at
adjacent time stamps are defined as temporal edges used to capture
interactions across different time stamps. Fig. 2 displays the spatial and
temporal edges. Although other nodes’ temporal edges are comparable, they are
not depicted in the illustration. Once the historical trajectories are
compiled into an integrated 3D graph, an adjacency matrix can be generated,
shown in Table I.

TABLE I: Adjacent matrix of biggraph

| A0 | B0 | C0 | D0 | A1 | B1 | C1 | D1 | A2 | B2 | C2 | D2
---|---|---|---|---|---|---|---|---|---|---|---|---
A0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0
B0 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0
C0 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0
D0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0
A1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1
B1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1
C1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 1
D1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 1 | 1
A2 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1
B2 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 1
C2 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0
D2 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0

###  IV-C Spatial-Temporal Fusion block

The model takes the location $(x_{i}^{t},y_{i}^{t})$ of traffic agents during
a historical period $T_{his}$ as input. To better represent this input, two
layers of MLP are utilized to increase its dimension. This results in an
output dimension of 16 for the MLP, as demonstrated in Equation (1).

$e_{i}^{t}=MLP(x_{i}^{t},y_{i}^{t}),where\ e_{i}^{t}\in\mathbb{R}^{F},F=16$
(1)

The output of the MLP is provided as input of two layers of GAT, allowing for
simultaneous capture of both spatial and temporal information. The input of
graph attention layers is Equation (2)

$h=\left\\{e_{1}^{1},e_{2}^{1},...,e_{N}^{1},e_{1}^{2},e_{2}^{2},...,e_{N}^{2},....,e_{1}^{T_{his}},e_{2}^{T_{his}},...,e_{N}^{T_{his}}\right\\}$
(2)

where $e_{n}^{t}$ is the embedding of $nth$ node at time $t$.

Figure 4: An illustration of graph attention layer. It allows a node to assign
different importance to different nodes within the threshold and in the
adjacent time stamps. The new feature of nodes is aggregated from them.

$\alpha_{u,v}^{i,j}=\frac{\exp\left(\text{LeakyRelu}\left(a^{T}\left[We_{u}^{i}\bigoplus
We_{v}^{j}\right]\right)\right)}{\sum_{k\in\mathbb{N}(u)}\sum_{l\in\left\\{i-1,i,i+1\right\\}}\exp\left(\text{LeakyRelu}\left(a^{T}\left[We_{u}^{i}\bigoplus
We_{k}^{l}\right]\right)\right)}$ (3)

where $\alpha_{u,v}^{i,j}$ is the attention weights between node $u$ at time
$i$ and node $v$ at time $j$. $W\in\mathbb{R}^{F^{\prime}\times F}$ is a
shared trainable matrix for linear mapping. $F^{\prime}$ is the output feature
dimension, and $F$ is the input feature dimension. ${\bigoplus}$ denotes the
concatenation operation. $\mathbb{N}(u)$ means the neighbor nodes of $u$
within a threshold at the same time. $a\in\mathbb{R}^{2F^{\prime}}$ is a
weight vector.

The updated representation of node $u$ at time $i$, denoted as
$e_{u}^{i^{\prime}}$, is obtained by computing the attention weights of
adjacent nodes on target nodes, as specified in Equation (4).

$e_{u}^{i^{\prime}}=\sum_{k\in\mathbb{N}(u)}\sum_{l\in\left\\{i-1,i,i+1\right\\}}\alpha_{u,v}^{i,j}We_{u}^{i}$
(4)

Figure 5: Spatial-Temporal fusion block

###  IV-D Spatial-Tempotal GCN block

The block records the historical $x,y$ coordinates of $N$ traffic agents and
processes spatial and temporal information separately. To capture
interactions, input data is first fed into a GCN layer, followed by a temporal
CNN layer to capture time information. The Spatial-Temporal GCN block consists
of three ST-GCN layers, as illustrated in Fig. (6).

Figure 6: Spatial-Tempotal GCN block

###  IV-E Trjectory prediction block

The trajectory prediction block is a conventional sequence-to-sequence
(seq2seq) model comprising GRUs as its components. The seq2seq model excels at
processing sequence data and has been widely utilized in prior studies such as
[4] and [22]. Previous experiments have demonstrated that the seq2seq model
exhibits high performance in trajectory prediction tasks. Furthermore,
trajectory prediction belongs to the domain of autoregressive tasks, which are
well-suited for LSTMs and GRUs. In this study, we employ the seq2seq model to
forecast future trajectories, building upon existing research.

##  V Experiments

We trained our model on a Linux server running Ubuntu 22.04.1 LTS with AMD
Ryzen Threadripper PRO 3995WX 64-Cores, 512GB RAM, and one NVIDIA GeForce RTX
3090.

###  V-A Dataset

We train and test our model on the ApolloScape Trajectory Dataset [12], whose
framerate are two frames per second. The training sequence lasts for 53
minutes, while the test sequence lasts for 50 minutes. The dataset contains
two files, one for training and one for testing. In the training zip file,
each file is a 1-minute sequence, and each line in the file contains frame
identification, object identification, object type, position x, position y,
position z, object length, object width, object height, and heading. The
traffic in the dataset is classified into five categories: small vehicles,
large vehicles, pedestrians, motorcyclists, bicyclists, and others. The
location is based on global coordinates. For the Apollo challenge, the
observation time was set to 2 seconds. However, some studies [9][23] have
extended the observation time to 3 seconds, along with a prediction time of 3
seconds. The testing files have the same data structure as the training files,
which are used to evaluate the performance of the model.

###  V-B Metrics

TABLE II: Competition Results On ApolloScape Trajectory Dataset.

Method | WSADE | $\text{ADE}_{\text{v}}$ | $\text{ADE}_{\text{p}}$ | $\text{ADE}_{\text{b}}$ | WSFDE | $\text{FDE}_{\text{v}}$ | $\text{FDE}_{\text{p}}$ | $\text{FDE}_{\text{b}}$
---|---|---|---|---|---|---|---|---
TafficPredict [12] | 8.5881 | 7.9467 | 7.1811 | 12.8805 | 24.2262 | 12.7757 | 11.1210 | 22.7912
GRIP++ [9] | 2.514 | 3.948 | 1.746 | 3.233 | 4.026 | 6.080 | 2.981 | 4.913
S2TNet [24] | 1.1679 | 1.9874 | 0.6834 | 1.700 | 2.1798 | 3.5783 | 1.3048 | 3.2151
STF | 1.384 | 2.403 | 0.799 | 2.001 | 1.707 | 2.972 | 1.012 | 2.392

Root Mean Square Error (RMSE): measures the distance between the predictor and
the ground truth at time $t$ [22].

$RMSE^{t}=\sqrt{\frac{\sum_{n\in\mathbb{N}}\left(\hat{y}_{n}^{t}-y_{n}^{t}\right)^{2}}{N}}$
(5)

The Apollo Scape Trajectory dataset provides a weighted sum formula for RMSE,
given by Equation (6).

$RMSE_{w}=D_{v}*RMSE_{v}+D_{p}*RMSE_{p}+D_{b}*RMSE_{b}$ (6)

where $D_{v}=0.20$,$D_{p}=0.58$, and $D_{b}=0.22$. ApolloScape Trajectory
Dataset provides these values and points out that they are computed based on
the velocity information available in the dataset. $RMSE_{v}$ represents the
RMSE value for vehicles, $RMSE_{p}$ represents the RMSE value for pedestrians,
and $RMSE_{b}$ represents the RMSE value for bikes.

Average Displacement Error (ADE): calculates the distance between the
predicted location of $N$ agents and ground truth in $T_{pred}$ time steps,
which measures the average prediction performance along the trajectory [8]

$ADE=\frac{\sum_{t\in\mathbb{T}}\sum_{n\in\mathbb{N}}\|\hat{y}_{n}^{t}-y_{n}^{t}\|_{2}}{N\times
T_{pred}}$ (7)

where $\hat{y}_{n}^{t}$ is the prediction location of $nth$ vehicle at time
step $t$, $y_{n}^{t}$ is the true location of $nth$ vehicle at time step $t$.
$\mathbb{T}=\\{T_{his}+1,T_{his}+2,...,T_{his}+T_{pred}\\}$ and
$\mathbb{N}=\\{1,2,...,N\\}$. $ADE_{v}$,$ADE_{p}$ and $ADE_{b}$ are computed
separately.

Final Displacement Error (FDE): only computes the distance between the
predicted final location and the true final location at the end of final
prediction time step $T_{pred}$ [8]

$FDE=\frac{\sum_{n\in\mathbb{N}}\|\hat{y}_{n}^{T_{pred}}-y_{n}^{T_{pred}}\|_{2}}{N}$
(8)

where $\hat{y}_{n}^{T_{pred}}$ is the prediction location of $nth$ vehicle at
the last prediction time step $T_{pred}$, $y_{n}^{T_{pred}}$ is the true
location of $nth$ traffic participant at the last prediction time step
$T_{pred}$. $FDE_{v}$, $FDE_{p}$ and $FDE_{b}$ are computed separately. The
WSADE and WSFDE can be computed through Equation (9) and Equation (10).

$WSADE=D_{c}*ADE_{v}+D_{p}*ADE_{p}+D_{b}*ADE_{b}$ (9)
$WSFDE=D_{c}*FDE_{v}+D_{p}*FDE_{p}+D_{b}*FDE_{b}$ (10)

where the $D_{v}=0.20$,$D_{p}=0.58$, and $D_{b}=0.22$ which is the same as
them in Equation (6) provided by the ApolloScape Trajectory Dataset.

###  V-C Experiments on the ApolloScape Trajectory Datasets

TABLE III: RMSE for trajectory prediction on Apollo Scape Trajectory

| Object type | RMSE
---|---|---
|  | 0.5s | 1s | 1.5s | 2s | 2.5s | 3s
| vehicle | 1.744 | 2.577 | 3.513 | 4.440 | 5.335 | 6.080
| pedestrian | 0.515 | 1.002 | 1.522 | 2.022 | 2.436 | 2.981
GRIP++ | bike | 1.101 | 2.145 | 2.924 | 3.779 | 4.535 | 4.913
| weighted RMSE | 0.890 | 1.569 | 2.228 | 2.892 | 3.478 | 4.026
| all objects | 1.384 | 2.218 | 3.033 | 3.919 | 4.722 | 5.425
| vehicle | 1.972 | 2.0 | 2.311 | 2.517 | 2.649 | 2.972
| pedestrian | 0.50 | 0.653 | 0.773 | 0.887 | 0.971 | 1.012
STF | bike | 1.366 | 1.833 | 1.925 | 2.207 | 2.282 | 2.392
| weighted RMSE | 0.985 | 1.182 | 1.334 | 1.503 | 1.595 | 1.707
| all objects | 1.586 | 1.792 | 2.029 | 2.265 | 2.363 | 2.612

Table II shows the competition results of TafficPredict, GRIP++, S2TNet, and
STF. The metrics reported include $WSADE$, $ADE_{v}$, $ADE_{p}$, $ADE_{b}$,
$WSFDE$, $FDE_{v}$, $FDE_{p}$, and $FDE_{b}$. Among the listed methods, it is
evident that the STF model demonstrates competitive performance across several
metrics. For the $WSADE$ metric, STF achieves a value of 1.384, which is lower
than TafficPredict (8.5881) and GRIP++ (2.514), indicating improved accuracy.
Similarly, for the $ADE_{v}$, $ADE_{p}$, and $ADE_{b}$ metrics, the STF model
outperforms TafficPredict and GRIP++, showing lower values and better accuracy
in trajectory prediction. But the higher values than the result of S2TNet,
which means the S2TNet has better performance than STF in short-time
prediction. Moving on to the $WSFDE$, $FDE_{v}$, $FDE_{p}$, and $FDE_{b}$
metrics, the STF model excels, demonstrating the lowest values among the
listed methods. Specifically, STF achieves an impressive $WSFDE$ of 1.707, an
$FDE_{v}$ of 2.972, an $FDE_{p}$ of 1.012, and an $FDE_{b}$ of 2.392. These
results indicate that the STF model can predict long-term trajectories with
greater precision and accuracy compared to TafficPredict, GRIP++, and S2TNet.

In summary, the STF model stands out as the best-performing method in
trajectory prediction based on the competition results. It surpasses
TafficPredict, GRIP++, and S2TNet in terms of $FDE$ metrics, demonstrating
superior accuracy and precision in trajectory prediction on long-time
trajectory prediction.

Due to the lack of $RMSE$ values at every frame in the TafficPredic and S2TNet
models, we evaluate our model by comparing it with GRIP++ based on the $RMSE$
at each prediction frame. Furthermore, our model incorporates GRIP++ as its
backbone. The outcomes derived from GRIP++ can be regarded as an ablation
experiment. The experiment results, as shown in Table III, compare the $RMSE$
for trajectory prediction on the Apollo Scape Trajectory dataset between the
GRIP++ model and the STF. The RMSE values are reported for different
prediction time intervals, ranging from 0.5s to 3s.

Overall, the STF outperforms the GRIP++ model in terms of trajectory
prediction accuracy. Specifically, the STF achieves lower $RMSE$ values for
most object types and prediction time intervals compared to GRIP++. For the
car object type, the STF obtains slightly higher $RMSE$ values at shorter
prediction time intervals (0.5s) but achieves better performance at longer
prediction time intervals (1s, 1.5s, 2s, 2.5s, and 3s). For pedestrians, the
STF consistently exhibits lower $RMSE$ values across all prediction time
intervals, especially for long-term prediction. For the bike object type, the
STF achieves better results at longer prediction time intervals (1s, 1.5s, 2s,
2.5s, and 3s), while the $RMSE$ values are comparable or slightly higher at
shorter intervals (0.5s). For the weighted sum of these three kinds of objects
computed according to Equation (6), the STF shows much better on the long-term
(1s, 1.5s, 2s, 2.5s, and 3s) trajectory prediction. The STF model demonstrates
superior performance compared to the GRIP++ model in terms of all object
types. This includes vehicles, pedestrians, bikes, and other objects. The STF
consistently achieves lower $RMSE$ values across long prediction time
intervals (1s, 1.5s, 2s, 2.5s, and 3s). Fig. 7 displays the plot of the
Weighted $RMSE$ for GRIP++ and STF at various predicted times. This figure
highlights the superior performance of STF in accurately predicting future
trajectories over an extended period.

Figure 7: Weighted RMSE in prediction time

##  VI Conclusion

This paper presents the STF model, designed to capture spatial and temporal
information from historical trajectory data simultaneously. The model
addresses the information missing problem in previous works while mitigating
the issue of error accumulation. Experiments are conducted on the ApolloScape
Trajectory Datasets. The results demonstrate superior performance compared to
previous models. Our model exhibits a significant advantage, particularly in
long-term trajectory prediction performance. Accurate long-term predictions
ensure that autonomous vehicles and robotics can proactively avoid potential
collisions, providing them with more time to respond, ultimately resulting in
enhanced safety.

## References

  * [1] W. Wang, L. Wang, C. Zhang, C. Liu, L. Sun _et al._ , “Social interactions for autonomous driving: A review and perspectives,” _Foundations and Trends® in Robotics_ , vol. 10, no. 3-4, pp. 198–376, 2022.
  * [2] A. Zhang, Z. C. Lipton, M. Li, and A. J. Smola, “Dive into deep learning,” _arXiv preprint arXiv:2106.11342_ , 2021.
  * [3] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, “Social gan: Socially acceptable trajectories with generative adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 2255–2264.
  * [4] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, “Social LSTM: Human Trajectory Prediction in Crowded Spaces,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. Las Vegas, NV, USA: IEEE, Jun. 2016, pp. 961–971.
  * [5] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel, “Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 424–14 432.
  * [6] C. Yu, X. Ma, J. Ren, H. Zhao, and S. Yi, “Spatio-temporal graph transformer networks for pedestrian trajectory prediction,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_. Springer, 2020, pp. 507–523.
  * [7] M. Mendieta and H. Tabkhi, “Carpe posterum: A convolutional approach for real-time pedestrian path prediction.” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 3, p. 2346–2354, Sep 2022. [Online]. Available: http://dx.doi.org/10.1609/aaai.v35i3.16335
  * [8] N. Deo and M. M. Trivedi, “Convolutional Social Pooling for Vehicle Trajectory Prediction,” May 2018.
  * [9] X. Li, X. Ying, and M. C. Chuah, “Grip++: Enhanced graph-based interaction-aware trajectory prediction for autonomous driving,” _arXiv preprint arXiv:1907.07792_ , 2019.
  * [10] Y. Zhu, D. Qian, D. Ren, and H. Xia, “Starnet: Pedestrian trajectory prediction using deep neural network in star topology,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 8075–8080.
  * [11] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [12] Y. Ma, X. Zhu, S. Zhang, R. Yang, W. Wang, and D. Manocha, “Trafficpredict: Trajectory prediction for heterogeneous traffic-agents,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 6120–6127.
  * [13] Z. Sheng, Y. Xu, S. Xue, and D. Li, “Graph-based spatial-temporal convolutional network for vehicle trajectory prediction in autonomous driving,” _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * [14] X. Mo, Y. Xing, and C. Lv, “Recog: A deep learning framework with heterogeneous graph for interaction-aware trajectory prediction,” _arXiv preprint arXiv:2012.05032_ , 2020.
  * [15] J. Sun, Q. Jiang, and C. Lu, “Recursive social behavior graph for trajectory prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 660–669.
  * [16] L. Shi, L. Wang, C. Long, S. Zhou, M. Zhou, Z. Niu, and G. Hua, “Sgcn: Sparse graph convolution network for pedestrian trajectory prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8994–9003.
  * [17] V. Kosaraju, A. Sadeghian, R. Martín-Martín, I. Reid, H. Rezatofighi, and S. Savarese, “Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [18] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun, “Learning lane graph representations for motion forecasting,” in _European Conference on Computer Vision_. Springer, 2020, pp. 541–556.
  * [19] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, “Vectornet: Encoding hd maps and agent dynamics from vectorized representation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 11 525–11 533.
  * [20] Z. Zhou, L. Ye, J. Wang, K. Wu, and K. Lu, “Hivt: Hierarchical vector transformer for multi-agent motion prediction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8823–8833.
  * [21] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [22] H. Song, W. Ding, Y. Chen, S. Shen, M. Y. Wang, and Q. Chen, “Pip: Planning-informed trajectory prediction for autonomous driving,” in _European Conference on Computer Vision_. Springer, 2020, pp. 598–614.
  * [23] X. Li, X. Ying, and M. C. Chuah, “Grip: Graph-based interaction-aware trajectory prediction,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 3960–3966.
  * [24] W. Chen, F. Wang, and H. Sun, “S2tnet: Spatio-temporal transformer networks for trajectory prediction in autonomous driving,” in _Asian Conference on Machine Learning_. PMLR, 2021, pp. 454–469.


# HybriMap: Hybrid Clues Utilization for Effective Vectorized HD Map
Construction

Chi Zhang 1 Qi Song1 Feifei Li1 Yongquan Chen2 &Rui Huang 1 1The Chinese
University of Hong Kong, Shenzhen
2Shenzhen Institute of Artificial Intelligence and Robotics for Society
{chizhang1, qisong<EMAIL_ADDRESS>{yqchen,
<EMAIL_ADDRESS>
###### Abstract

Constructing vectorized high-definition maps from surround-view cameras has
garnered significant attention in recent years. However, the commonly employed
multi-stage sequential workflow in prevailing approaches often leads to the
loss of early-stage information, particularly in perspective-view features.
Usually, such loss is observed as an instance missing or shape mismatching in
the final birds-eye-view predictions. To address this concern, we propose a
novel approach, namely HybriMap, which effectively exploits clues from hybrid
features to ensure the delivery of valuable information. Specifically, we
design the Dual Enhancement Module, to enable both explicit integration and
implicit modification under the guidance of hybrid features. Additionally, the
perspective keypoints are utilized as supervision, further directing the
feature enhancement process. Extensive experiments conducted on existing
benchmarks have demonstrated the state-of-the-art performance of our proposed
approach.

##  1 Introduction

The vectorized High-Definition (HD) map construction under Birds-Eye View
(BEV) has been widely investigated in recent autonomous driving studies, due
to the extensive demand in motion panning Jian et al. (2019); Diaz-Diaz et al.
(2022) and forecasting Gao et al. (2020); Da and Zhang (2022); Conde et al.
(2023). In particular, vectorized HD maps represent static map elements, such
as pedestrian crossing, lane divider, and road boundaries, using parametric
curves Qiao et al. (2023) or 2D vectors Li et al. (2022); Liu et al. (2023);
Liao et al. (2022); Ding et al. (2023); Liao et al. (2023), providing notable
benefits in terms of scalability and storage consumption. In this paper, we
continue representing map elements in 2D vectors and direct our efforts toward
enhancing the precision and reliability of vectorized HD map construction.

Figure 1: Hard cases in HD map construction. MapTRv2Liao et al. (2023) tends
to lose specific instance clues or shape priors existing in the input images,
as visualized in (b). In the proposed HybriMap, we can well maintain the given
priors and thus strengthen the alignment between the original input and map
prediction, by employing a hybrid approach.

Given the fact that input images and output map predictions exist in separate
coordinate systems, existing HD map construction methods, such as HDMapNet Li
et al. (2022), VectorMapNet Liu et al. (2023), and MapTRv2 Liao et al. (2023),
typically employ a sequential workflow incorporating multi-stage processes: 1)
an image backbone network to extract perspective-view (PV) features from input
images; 2) a cross-view transformation network to project PV features into the
birds-eye view; 3) a transformer-based BEV decoder to generate the final map
predictions. However, the sophisticated nature of the cross-view
transformation network introduces a significant challenge in terms of early-
stage data loss, specifically the PV features. Taking MapTRv2 as an example,
such a prior loss problem is often observed as the discrepancies between the
input PV images and the final map predictions, as presented in Fig.1 (b). For
instance, the lane divider highlighted in the blue area is visible in the left
image but fails to be detected in the MapTRv2. Additionally, the road boundary
indicated by the yellow circle appears to extend beyond the crossing in the
right image but is mispredicted as a small instance by MapTRv2.

Intuitively, the aforementioned issue of early-stage data loss can be
effectively addressed by reusing PV features that are full of spatial priors.
Nevertheless, it is worth noting that PV features often contain a large amount
of redundant information from irrelevant categories like the sky and
buildings, as only map elements are necessary for BEV map prediction.
Conversely, despite the BEV representations produced by the cross-view
transformation network suffering from prior loss, they still prioritize and
highlight more meaningful and intricate elements compared to PV features.
Consequently, a new question arises: How can we efficiently preserve the
genuinely useful clues contained within both PV and BEV features?

We found that both PV feature and BEV feature can serve as valuable references
or complementary sources of each other during feature enhancement. This
motivates us to propose the Hybrid approach of HD Map construction (HybriMap),
which involves a parallel channel to implement comprehensive and adequate
exploitation of features from both PV and BEV into effective HD map
construction. Specifically, we design a Dual Enhancement Module (DEM) to
achieve feature enhancement through both explicit integration of hybrid
features, and implicit upgrading of BEV features. The explicit branch of the
DEM fuses features from both perspectives to capture an Integrated Positional
Embedding and assist the subsequent decoding. Simultaneously, the implicit
branch of the DEM focuses on emphasizing the valuable clues within the BEV
features, taking advantage of the preceding fusion process that extracts the
most worthwhile portions from the BEV features. This dual approach ensures a
comprehensive and effective enhancement of the overall feature set.

Unlike the previous sequential workflow, which heavily depends on BEV
representation, our proposed HybriMap leverages hybrid features to their
fullest extent, addressing the issue of early-stage information loss. As
illustrated in Fig.1 (c), HybriMap demonstrates its ability to preserve the
initial priors from the original images and efficiently convert them into
vectorized map instances, surpassing the performance of sequential pipelines
like MapTRv2.

The contributions of this work are as follows:

  * •

We propose a Hybrid approach, which enables the re-utilization of PV features
and interaction between features from both perspectives in vectorized HD map
construction.

  * •

We introduced the Dual Enhancement Module (DEM) that sufficiently leverages
hybrid information through explicit integration and implicit upgrading.

  * •

Our approach exhibits state-of-the-art results on the existing benchmarks,
particularly with a notable improvement of 3.9% mAP on nuScenes dataset.

##  2 Related Work

###  2.1 Vectorized HD Map Construction

HDMapNet Li et al. (2022) involves a post-processing procedure into the
semantic segmentation and instance embedding, enabling the map to represent
static elements in a vectorized form. Subsequent studies commonly employ the
end-to-end networks to attain HD map construction. Several methods use the
Bezier curve to represent map instances and predict HD map via parametric
curve estimation Can et al. (2022); Qiao et al. (2023). One alternative is to
represent map elements as 2D vectors. VecterMapNet Liu et al. (2023)
structures the map construction as a vector detection problem. MapTR Liao et
al. (2022) completes the map vector representation and refines the overall
detection pipeline in a more concrete manner. Recent methods Ding et al.
(2023); Yu et al. (2023); Liao et al. (2023); Yuan et al. (2024) generally
follow the conventional sequential process of 1) PV feature extraction, 2) BEV
feature transformation, and 3) map vector prediction.

However, map predictions through the sequential process are vulnerable to
early-stage information loss. Differently, we propose a novel Hybrid approach
that re-utilizes PV features based on the hybrid of PV and BEV features, to
address the issue of information loss that commonly occurs in traditional
pipelines.

###  2.2 Cross-View Information Utilization

The utilization of cross-view information is a typical topic in the map-
learning area. PyOccNet Roddick and Cipolla (2020) transforms PV features into
BEV through a pyramid network, preserving multi-scale information from inputs.
The existing Cross-view Transformation Yang et al. (2021) activates the mutual
response between cross-view features, to enhance the feature in the objective
domain. Zhou et al. Zhou and Krähenbühl (2022) involve image-view features and
camera-aware embedding in the implementation of segmentation under map-view.
BEVFormerV2 Yang et al. (2023) simply performs post-processing and encoding on
PV features and thus enhances the network’s decoding capability for BEV
features. ScalableMap Yu et al. (2023) employs a fusion module on two branches
originating from PV features to enhance the BEV features.

In our HybriMap, we design the Dual Enhancement Module, where the implicit and
explicit cross-view utilization are coupled. DEM simultaneously activates the
implicit enhancement of BEV representation and elevates the decoding
performance of map predictions through explicit aggregation of hybrid
features.

##  3 Method

###  3.1 Architecture Overview

The pipeline of HybriMap is depicted in Fig. 2. Given a set of RGB images
$\\{I_{i}\in\mathbb{R}^{H_{I}\times W_{I}\times 3},i=1,2,...,N\\}$ from $N$
cameras, HybriMap aims to predict $N_{m}$ map instances on $N_{c}$ classes in
the form of vectors with $N_{p}$ points
$\\{\hat{v}_{i}\in\mathbb{R}^{N_{p}\times 2},i=1,2,...,N_{m}\\}$ and the
classification scores $\\{\hat{c}_{i}\in\mathbb{R}^{N_{c}\times
1},i=1,2,...,N_{m}\\}$ eventually.

Figure 2: The overall architecture of HybriMap. The DEM is depicted in an
abstract version. ”$\mathcal{L}$” indicates that the outputs are supervised by
the corresponding loss, and ”PE” means positional embedding. Different from
the typical sequential workflow, the proposed method leverages a Hybrid
approach that deeply exploits features from both PV and BEV. Such an approach
enhances the reliance of the prediction results on the original information,
thereby mitigating the negative impacts of early-stage information loss.

##### Image Backbone.

The input images are sent into the image backbone and converted to PV features
$\\{F^{P}_{i}\in\mathbb{R}^{H_{F}\times W_{F}\times C_{F}},i=1,2,...,N\\}$. By
default, we adopt ResNet He et al. (2016) as the image backbone.

##### PV-to-BEV.

The PV-to-BEV module further transforms the PV features into BEV features
$F^{B}\in\mathbb{R}^{H\times W\times C}$. LSS transform Philion and Fidler
(2020) is employed to be the default method.

##### Dual Enhancement Module.

In the Dual Enhancement Module, the hybrid features consisting of PV features
$F^{P}$ and BEV features $F^{B}$ are utilized as inputs. Specifically, $F^{P}$
and $F^{B}$ are fused to generate Integrated Heatmap $H^{P}$, which is then
converted into Integrated Positional Embedding $E_{I}$. $B$ is synchronously
updated as implicitly enhanced BEV feature $F^{B}_{E}$. The thorough purpose
and operation of DEM will be introduced in Sec. 3.2.

##### Map Element Decoder.

Outputs from DEM, i.e. $E_{I}$ and $F^{B}_{E}$ are then transferred into a
transformer-based map element decoder. We leverage the transformer-based
decoder Carion et al. (2020); Zhu et al. (2020) to directly output the
prediction results. Specifically, we adopt hierarchical query $Q$ Liao et al.
(2022) and enhanced BEV features $F^{B}_{E}$ as the query and value of the
decoder, respectively. The positional embedding used in the decoder is
combined by the globally applicable learnable positional embedding Gehring et
al. (2017), and the Integrated Positional Embedding $E_{I}$ that is relevant
to the hybrid features. The decoder directly predicts $N_{m}$ map elements,
containing coordinates $\hat{v}$ and classification score $\hat{c}$ of the
vectors.

##### Losses.

We propose Heatmap Loss $\mathcal{L}_{H}$ to constrain the enhancement effects
of DEM. Additionally, Rasterized Instance Segmentation (RIS) Loss
$\mathcal{L}_{RIS}$ is introduced to further improve the prediction
performance. The details of the loss components are further discussed in
Sec.3.3.

###  3.2 Dual Enhancement Module

Figure 3: Workflow of the Dual Enhancement Module. Given the hybrid
information, Integrated Positional Embedding explicitly incorporates features
from both perspectives, where the intermediate Integrated Heatmap is
supervised. BEV features, based on the extracted useful portion among itself,
are thereby implicitly enhanced. The dual enhancement enables the sufficient
and effective utilization of hybrid information.

The Dual Enhancement Module is designed for the deep exploitation of hybrid
features, as depicted in Fig. 3. The valuable priors within the hybrid
features are explicitly integrated and formed as position embedding. To
facilitate the efficacy, a crucial intermediate, i.e., Integrated Heatmap, is
supervised. Additionally, the extracted refined BEV features are further
leveraged for the implicit enhancement of themselves. Such enhancement
highlights the genuinely useful portions within the hybrid information and
guides the subsequent decoding process.

Figure 4: Illustration of Integrated Heatmap and BEV Enhancement. ”Enh.”
indicates the implicit enhancement process on the BEV features. As presented
in (a), the integrated heatmap gathers higher heat intensities around the
crucial map instances, e.g. pedestrian crossing and road boundaries.
Additionally, the second row shows the BEV feature maps before and after
implicit enhancement. It is evident that the original BEV features in (b)
basically depict the distinction of the drivable area, whereas the enhanced
BEV features in (c) accentuate the details near map instances.

####  3.2.1 Integrated Positional Embedding

The PV and BEV information is explicitly incorporated to generate the
Integrated Positional Embedding in this branch of DEM. According to the design
of the map decoder Zhu et al. (2020), the positional embedding serves not only
as a decorative element for the input information but also plays a role in
generating reference points, which serve as the initial estimates of the map
vectors. By incorporating hybrid features into the positional embedding, the
decoded feature representation and vector estimation become more receptive to
the cues derived from the hybrid information, as evident from the functional
implications of the positional embedding.

Suppose $\Psi^{n}(\cdot)$ is the convolution network function connected with
$n$ cascaded layers. We first downsample $F^{B}$ into the BEV Enhancement Map
$F^{B}_{M}$ from BEV features by multi-layer convolution:

$F^{B}_{M}=\Psi^{3}(F^{B})$ (1)

where $F^{B}_{M}\in\mathbb{R}^{(H/d)\times(W/d)\times C}$, $d$ is the pre-
defined downsample scale factor.

$F^{B}_{M}$ is then projected into perspective-view. For an arbitrary
coordinate $(x,y)$ in the BEV plane, we first complete it to a 3D coordinates
$(x,y,z)$ by assigning a constant height value. Given the camera parameter
matrix $\boldsymbol{P}$, the coordinate $(x,y,z)$ in the 3D camera ego space
can be converted to pixel $(u,v)$ in the image coordinate by the following
relationship:

$[u,v,1]^{T}=\boldsymbol{P}\times[x,y,z,1]^{T}$ (2)

We create a perspective-view feature map for each class, denoted as
$F^{P}_{M}\in\mathbb{R}^{H_{I}\times W_{I}\times N_{c}}$ and initialize it to
be blank. The BEV Enhancement Map $F^{B}_{M}$ generated from Eq.1 is firstly
encoded by multilayer perceptron ($MLP$) and projected by assigning feature
values to the corresponding region:

$F^{P}_{M}[u,v,c^{id}]=MLP(F^{B}_{M})[x,y,c^{id}]$ (3)

in which the relationship between $(u,v)$ and $(x,y)$ is illustrated in Eq. 2,
and $c^{id}$ is the class index. The Integrated Heatmap $H^{P}$ is obtained by
aggregating the PV features and the projected perspective-view feature map:

$H^{P}=MLP(F)+F^{P}_{M}$ (4)

It is worth noting that $F_{H}$ is supervised in Heatmap Loss, which will be
illustrated in 3.3.

Ultimately, the Integrated Positional Embedding $E_{I}$ is encoded from the
Integrated Heatmap:

$E_{I}=MLP(H^{P})$ (5)

As a result, $E_{I}$ is imbued with hybrid cues, enabling it to effectively
enhance the subsequent decoding performance.

The visualized samples of Integrated Heatmap in Fig.4 (a) indicate a strong
correlation between the distribution of heat intensities and the map instances
in the perspective view. For example, higher heat intensities occur near the
pedestrian crossing in the left figure and the road boundaries in both
figures. Since Integrated Positional Embedding is directly converted from the
Integrated Heatmap, it inherently encompasses dependable fused hybrid priors.
Therefore, integrating the hybrid features into the formation of positional
embedding enforces the decoding to be guided by the high-quality priors from
both perspectives, leading to advanced decoding performance on the BEV
features.

####  3.2.2 Enhancement BEV Feature

Within the framework of DEM, the implicit enhancement of BEV features is
accomplished through the utilization of the BEV Enhancement Map $F^{B}_{M}$,
derived from the BEV features themselves. However, owing to the intrinsic
relationship between the enhancement map and the supervised Integrated
Heatmap, the BEV Enhancement Map $F^{B}_{M}$ undergoes indirect influence,
deliberately reinforcing the representation of BEV features. Accordingly, the
BEV features are effectively enhanced in an indirect manner, guided by the
integration of hybrid inputs.

The implicit enhancement on BEV features is exclusively carried out within the
domain of BEV. Given the BEV Enhancement Map derived from Eq. 1, we have the
Enhanced BEV features $F^{B}_{E}$ by collapsing the $F^{B}$ and $F^{B}_{M}$:

$F^{B}_{E}=\Psi^{1}(Cat(F^{B},\Omega(F^{B}_{M})))$ (6)

where $\Omega(\cdot)$ is the upsample function and ”$Cat$” indicates
concatenation along the feature channel. The Enhanced BEV features
$F^{B}_{E}\in\mathbb{R}^{H\times W\times C}$ eventually serve as the input
feature of the map element decoder.

Fig.4 (b) and (c) visually depict the disparities in the representation of BEV
features before and after the implicit enhancement. Compared to the original
BEV features in (b), the enhanced BEV features in (c) showcase a more
emphasized depiction of map instance information, particularly near the road
boundaries and the pedestrian crossing. In terms of results, this selective
emphasis on the BEV features through implicit enhancement essentially yields a
representation that aligns more closely with the distribution of practical map
instances.

###  3.3 Training Loss

##### Heatmap Loss.

The objective of Heatmap Loss involvement is to deliberately constrain the
enhancement process within DEM. As introduced in Sec. 3.2, the Integrated
Heatmap $H^{P}$ is a crucial role in DEM, given its close association with
both Integrated Position Embedding and the BEV Enhancement Map. To guarantee
the appropriate influence of the hybrid information on $H^{P}$, it is
supervised by the heatmap formed by the keypoints present in the map vectors.

Given the annotated keypoints projected on perspective-view, the generation of
target heatmap $H^{P}_{T}$ basically follows the keypoint heatmap in previous
object detection methods Law and Deng (2018); Duan et al. (2019).
Specifically, we cast all groundtruth keypoints using a Gaussian kernel with
constant deviation. The loss $\mathcal{L}_{H}$ is computed by:

$\mathcal{L}_{H}=\mathcal{L}_{Focal}(Sigmoid(H^{P}),H^{P}_{T})$ (7)

where $\mathcal{L}_{Focal}$ is the focal loss Lin et al. (2017) between the
predicted heatmap and the target.

##### Rasterized Instance Segmentation Loss.

To balance the effects of keypoint emphasis and avoid neglect of trivial
points, we introduce the Rasterized Instance Segmentation (RIS) Loss
$\mathcal{L}_{RIS}$ into the overall supervision. For a predicted vector
$\hat{v}_{i}$, we assign the classification score $\hat{c}_{i}$ to the
corresponding pixels to form a BEV mask $M_{R}$, while the binary target mask
$T_{R}$ is depicted by groundtruth class labels, represented as one-hot vector
$c_{i}$. The RIS loss is the cross-entropy loss between the BEV mask and the
target mask:

$\mathcal{L}_{RIS}=\mathcal{L}_{CE}(M^{R},M^{R}_{T})$ (8)

##### Total Loss.

Following the previous study Liao et al. (2022), we leverage focal loss to
supervise the category of the predicted vectors, and L1-norm for geometrical
properties, whose results are denoted as classification loss
$\mathcal{L}_{cls}$ and point-to-point loss $\mathcal{L}_{pts}$, respectively.

Besides, we inherit the loss combination $\mathcal{L}_{BS}$ from the baseline
method Liao et al. (2023), involving in edge direction loss, auxiliary
one2many loss, and auxiliary dense loss. The total loss $\mathcal{L}_{total}$
that participates in the training of HybriMap is

$\mathcal{L}_{total}=\gamma_{H}\mathcal{L}_{H}+\gamma_{R}\mathcal{L}_{RIS}+\mathcal{L}_{cls}+\mathcal{L}_{pts}+\mathcal{L}_{BS}$
(9)

where $\gamma_{H}$ and $\gamma_{R}$ are the weighted factors.

Method | Backbone | Epochs | General Thresholds | Tighter Thresholds
---|---|---|---|---
$AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$ | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
HDMapNet Li et al. (2022) | EB0 | 30 | 14.4 | 21.7 | 33.0 | 23.0 | 7.1 | 28.3 | 32.6 | 22.7
VectormapNet Liu et al. (2023) | R50 | 110 | 36.1 | 47.3 | 39.3 | 40.9 | 18.2† | 27.2† | 18.4† | 21.3†
MapTR Liao et al. (2022) | R50 | 24 | 46.3 | 51.5 | 53.1 | 50.3 | 23.2† | 30.7† | 28.2† | 27.3†
BeMapNet Qiao et al. (2023) | R50 | 30 | 57.7 | 62.3 | 59.4 | 59.8 | 39.0 | 46.9 | 37.8 | 41.3
PivotNet Ding et al. (2023) | R50 | 24 | 56.2 | 56.5 | 60.1 | 57.6 | 34.3 | 41.4 | 39.8 | 38.5
MapTRv2 Liao et al. (2023) | R50 | 24 | 59.8 | 62.4 | 62.4 | 61.5 | 34.1† | 41.6† | 36.8† | 37.5†
HybriMap | R50 | 24 | 63.0 | 65.9 | 67.2 | 65.4 | 38.1 | 45.7 | 42.1 | 41.9
Table 1: Comparison with SOTA methods on nuScenes validation set. ”EB0” is the
backbone of previous work Tan and Le (2019). ”†” indicates the results are re-
evaluated by publicly available models. The best and second best results are
highlighted in bold and underline, respectively.

##  4 Experiments

###  4.1 Experimental Settings

##### Dataset.

We evaluate our method on two popular benchmarks: nuScenesCaesar et al. (2020)
and Argoverse 2 Wilson et al. (2021). Each dataset comprises 1000 scenes in
total, which are divided into train/validation/test sets with a split of
700/150/150 scenes. In nuScenes, each scene contains a video approximately 20s
long, where key frames are annotated at 2Hz. These key frames are all captured
by 6 surround view onboard cameras with 360 degrees field-of-view, containing
6 images for each. Argoverse 2 dataset annotates key frames at 10Hz, in which
the key frame contains 7 ring cameras.

##### Metric.

Following the previous works Liu et al. (2023); Liao et al. (2022), we use
Average Precision (AP) to evaluate the performance of the methods. A predicted
instance is defined as true positive only if its Chamfer Distance from the
groundtruth is less than a certain threshold. The thresholds are generally
defined as $\\{0.5m,1.0m,1.5m\\}$. Meanwhile, we support tighter thresholds
$\\{0.2m,0.5m,1.0m\\}$ for a comprehensive comparison with a more precise
scale.

##### Implementation Details.

The map instances involves static elements from 3 classes, i.e. pedestrian
crossing (ped), lane divider (div), and road boundary (bdr). The perceptual
range of the map is defined as the area spanning $15m$ left and right, as well
as $30m$ front and rear from the ego vehicle. The amount of map vector
predictions $N_{m}$ is set to be 50 and each vector contains 20 points on the
BEV plane. The weights of the proposed losses are empirically are set as
$\gamma_{H}=0.1$ and $\gamma_{RIS}=15$. Regarding the network, we adopt
ResNet50 (R50) He et al. (2016) as the image backbone and AdamW Loshchilov and
Hutter (2018) as the optimizer.

###  4.2 Comparisons with State-of-the-art Methods

##### Results on nuScenes.

We compare our results with the existing methods on nuScenes dataset, as
listed in Tab. 1. Taking results on general thresholds as an example, we
achieved appreciable improvements, surpassing the SOTAs by $3.9\%$ mAP with
24-epoch training sessions. The results exhibit that our method achieves not
only state-of-the-art performance on both threshold sets but also a
significantly faster converging speed.

In terms of the obtained results, our method generally demonstrates the most
notable improvements in the road boundary class compared to existing methods.
This improvement can be attributed to the exploitation of perspective priors
to its full extent, as within the three static map elements, roadsides
typically exhibit not only prominent color variations but are also surrounded
by noticeable ground condition changes, thereby possessing the richest clues
from the original inputs.

##### Results on Argoverse 2.

To further demonstrate the wide effectiveness of our approach, we present the
experimental results on the larger Argoverse 2, as shown in Tab. 2. HybriMap
gains $2.5\%$ and $5.2\%$ improvement in mAP beyond the existing methods, for
the two threshold sets respectively. As demonstrated by the experimental
results, our method consistently achieves more advanced results in map
prediction, both under general and more stringent criteria.

##### Qualitative Analysis.

We present the visualized map construction results under various weather or
illumination conditions in Fig. 5. The areas in yellow highlight the
discrepancy between map predictions and the corresponding perspective-view
clues. Compared with the existing method based on the conventional sequential
framework, our HybriMap successfully detects the partition and the sideway
along the roadside (e.g. Row 1 and Row 3) as well as the faraway crossing
(e.g. Row 2), indicating a more effective utilization of perspective-view
hints.

Moreover, due to the mechanism of Heatmap Loss, which supervises the keypoint
distribution under perspective-view, our methods can address the issue of
incomplete original information caused by occlusion or image blur, to some
extent. As shown in Fig. 5 Row 2, the lane divider in the blue shade area is
nearly invisible in the raw input. With the combined effect of Integrated
Heatmap supervision and feature enhancement, instances in this region have
been emphasized and successfully decoded.

However, due to the strong dependence on the original inputs, our approach is
limited under the circumstances that PV features are even difficult to
extract. For example in Fig. 5 Row 3, the sideway is mispredicted with some
directional offset.

Method | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
---|---|---|---|---
VectormapNet Liu et al. (2023) | 38.3 | 36.1 | 39.2 | 37.9
MapTRv2 Liao et al. (2023) | 62.9 | 72.1 | 67.1 | 67.4
HybriMap | 66.6 | 73.0 | 70.0 | 69.9
PivotNet Ding et al. (2023) | 31.3 | 47.5 | 43.4 | 40.7
HybriMap | 37.9 | 55.4 | 44.3 | 45.9
Table 2: Comparison with SOTA methods on Argoverse 2 validation set. All the
experiment above contains 6-epoch training based on R50 backbone. Results in
the top section are evaluated on general thresholds, while in bottom section
use tighter thresholds. Figure 5: Qualitative results on nuScenes validation
dataset. We provide the complete surround view inputs, predictions of
MapTRv2Liao et al. (2023), HybriMap, and the ground truth map. The regions
that are highlighted by colored ellipses enclose instances that are difficult
to detect.

###  4.3 Ablation Studies

Ablation experiments are conducted in this section to analyze the effects of
individual modules, including the hybridization of features, DEM, and various
loss components. The experiments in this section are trained with R50 backbone
through 24 epochs, and evaluated on nuScenes validation set on the general
threshold set.

##### Effectiveness of Hybrid Features.

In this section, we aim to specify the effectiveness of hybrid features
through comparative experiments. Concerning the fact that the implicit
enhancement on BEV features themselves is depended on the hybrid features, we
take positional embedding as the example to compare the impact of encoding
based on pure PV features or hybrid features on the map estimation results. As
listed in Tab. 3, the positional embedding encoded by hybrid features achieves
$0.9\%$ mAP higher than that from pure PV features. The results demonstrate
that the involvement of BEV features is capable of providing significant
referential information, regarding the re-utilization of PV features.

PE Mode | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
---|---|---|---|---
PV | 60.9 | 62.8 | 66.2 | 63.3
Hybrid | 61.9 | 64.7 | 66.0 | 64.2
Table 3: Comparison between pure PV features and Hybrid features regarding
positional embedding mode. The column ”PE Mode” indicates the positional
embedding is encoded by pure PV features (Row 1) or hybrid features (Row 2).

##### Effectiveness of Dual Enhancement Module.

In order to assess the efficacy of the dual enhancement, we perform individual
experiments on the two outputs generated by the DEM, namely Integrated
Positional Embedding and implicitly Enhanced BEV features. The experimental
results, presented in Tab. 4, demonstrate the impact of explicit integration
on positional embedding and BEV enhancement. These enhancements lead to
improvements of $1.9\%$ and $1.0\%$ in mAP, respectively, in the final
results. Moreover, the dual enhancement can leverage the advantages of both
enhancement branches, resulting in a significant mAP improvement of $3.1\%$.

Exp. | Int. | Enh. | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
---|---|---|---|---|---|---
0 |  |  | 59.7 | 62.3 | 64.9 | 62.3
1 | ✓ |  | 61.9 | 64.7 | 66.0 | 64.2
2 |  | ✓ | 60.9 | 63.5 | 65.5 | 63.3
3 | ✓ | ✓ | 63.0 | 65.9 | 67.2 | 65.4
Table 4: Ablation studies of DEM. ”Int.” and ”Enh.” represent the
participation of Integrated Positional Embedding and implicit Enhancement on
BEV features, respectively.

##### Effectiveness of Loss Components.

We further verify the impacts of the proposed losses, i.e. Heatmap Loss and
RIS Loss, as demonstrated in Tab. 5. Under the joint influence of
$\mathcal{L}_{H}$ and $\mathcal{L}_{RIS}$, the overall results attain a
$1.7\%$ increase in mAP.

Exp. | $\mathcal{L}_{H}$ | $\mathcal{L}_{RIS}$ | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
---|---|---|---|---|---|---
0 |  |  | 62.1 | 64.0 | 65.0 | 63.7
1 | ✓ |  | 62.8 | 65.0 | 65.5 | 64.4
2 |  | ✓ | 60.9 | 65.2 | 66.7 | 64.3
3 | ✓ | ✓ | 63.0 | 65.9 | 67.2 | 65.4
Table 5: Ablation studies of loss components.

###  4.4 Extended task: 3D Map Construction

HybriMap demonstrates its capability in 3D map construction by elevating the
dimension of map vectors to 3-dimensional space. The experiment on 3D map
construction is presented in Table 6, conducted on the validation set of
Argoverse 2, with evaluation based on AP using general thresholds. The results
clearly indicate a notable improvement ($3.8\%$ mAP) achieved by our method.
This significant advancement can be attributed to the influential role of PV
features, which facilitate the integration of height information into BEV
features and enable their effective decoding as 3D map vectors. The results of
this experiment indicate that our method can maintain its superiority in the
derivative task.

Method | $AP_{ped}$ | $AP_{div}$ | $AP_{bdr}$ | $mAP$
---|---|---|---|---
VectormapNet Liu et al. (2023) | 36.5 | 35.0 | 36.2 | 35.8
MapTRv2 Liao et al. (2023) | 60.7 | 68.9 | 64.5 | 64.7
HybriMap | 64.0 | 72.5 | 69.1 | 68.5
Table 6: Experimental results of 3D map construction. Our HybridMap framework
exhibits applicability in 3D map vector prediction, showcasing significant
advancements compared to previous studies.

##  5 Conclusion

In this paper, we propose HybriMap to address the issue of information loss in
HD map construction. In particular, we design a hybrid approach to effectively
re-utilize the perspective-view features from the original inputs.
Furthermore, the Dual Enhancement Module is employed to enhance both the
decoding performance and the BEV representation, simultaneously. Extensive
experiments on the dataset have demonstrated the superiority of our method,
achieving SOTA performance in HD map construction. Furthermore, our method can
be effectively applied to the derivative task. We anticipate that the
aforementioned research can provide more reliable map support for autonomous
driving and contribute to future investigations in HD map studies.

## References

  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020.
  * Can et al. [2022] Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool.  Topology preserving local road network estimation from single onboard camera image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17263–17272, 2022.
  * Carion et al. [2020] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * Conde et al. [2023] Marcos V Conde, Rafael Barea, Luis M Bergasa, and Carlos Gómez-Huélamo.  Improving multi-agent motion prediction with heuristic goals and motion refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5322–5331, 2023.
  * Da and Zhang [2022] Fang Da and Yu Zhang.  Path-aware graph attention for hd maps in motion prediction.  In 2022 International Conference on Robotics and Automation (ICRA), pages 6430–6436. IEEE, 2022.
  * Diaz-Diaz et al. [2022] Alejandro Diaz-Diaz, Manuel Ocaña, Ángel Llamazares, Carlos Gómez-Huélamo, Pedro Revenga, and Luis M Bergasa.  Hd maps: Exploiting opendrive potential for path planning and map monitoring.  In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1211–1217. IEEE, 2022.
  * Ding et al. [2023] Wenjie Ding, Limeng Qiao, Xi Qiu, and Chi Zhang.  Pivotnet: Vectorized pivot learning for end-to-end hd map construction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3672–3682, October 2023.
  * Duan et al. [2019] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019.
  * Gao et al. [2020] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid.  Vectornet: Encoding hd maps and agent dynamics from vectorized representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11525–11533, 2020.
  * Gehring et al. [2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In International conference on machine learning, pages 1243–1252. PMLR, 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * Jian et al. [2019] Zhiqiang Jian, Songyi Zhang, Shitao Chen, Xin Lv, and Nanning Zheng.  High-definition map combined local motion planning and obstacle avoidance for autonomous driving.  In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 2180–2186. IEEE, 2019.
  * Law and Deng [2018] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In Proceedings of the European conference on computer vision (ECCV), pages 734–750, 2018.
  * Li et al. [2022] Qi Li, Yue Wang, Yilun Wang, and Hang Zhao.  Hdmapnet: An online hd map construction and evaluation framework.  In 2022 International Conference on Robotics and Automation (ICRA), pages 4628–4634. IEEE, 2022.
  * Liao et al. [2022] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang.  Maptr: Structured modeling and learning for online vectorized hd map construction.  In The Eleventh International Conference on Learning Representations, 2022.
  * Liao et al. [2023] Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang.  Maptrv2: An end-to-end framework for online vectorized hd map construction.  arXiv preprint arXiv:2308.05736, 2023.
  * Lin et al. [2017] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * Liu et al. [2023] Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao.  Vectormapnet: End-to-end vectorized hd map learning.  In International Conference on Machine Learning, pages 22352–22369. PMLR, 2023.
  * Loshchilov and Hutter [2018] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * Philion and Fidler [2020] Jonah Philion and Sanja Fidler.  Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d.  In Proceedings of the European Conference on Computer Vision, 2020.
  * Qiao et al. [2023] Limeng Qiao, Wenjie Ding, Xi Qiu, and Chi Zhang.  End-to-end vectorized hd-map construction with piecewise bezier curve.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13218–13228, 2023.
  * Roddick and Cipolla [2020] Thomas Roddick and Roberto Cipolla.  Predicting Semantic Map Representations From Images Using Pyramid Occupancy Networks.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11135–11144, Seattle, WA, USA, June 2020. IEEE.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, June 2019.
  * Wilson et al. [2021] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays.  Argoverse 2: Next generation datasets for self-driving perception and forecasting.  In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021), 2021.
  * Yang et al. [2021] Weixiang Yang, Qi Li, Wenxi Liu, Yuanlong Yu, Yuexin Ma, Shengfeng He, and Jia Pan.  Projecting your view attentively: Monocular road scene layout estimation via cross-view transformation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15536–15545, 2021.
  * Yang et al. [2023] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al.  Bevformer v2: Adapting modern image backbones to bird’s-eye-view recognition via perspective supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17830–17839, 2023.
  * Yu et al. [2023] Jingyi Yu, Zizhao Zhang, Shengfu Xia, and Jizhang Sang.  Scalablemap: Scalable map learning for online long-range vectorized hd map construction.  In 7th Annual Conference on Robot Learning, 2023.
  * Yuan et al. [2024] Tianyuan Yuan, Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao.  Streammapnet: Streaming mapping network for vectorized online hd map construction.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 7356–7365, 2024.
  * Zhou and Krähenbühl [2022] Brady Zhou and Philipp Krähenbühl.  Cross-view transformers for real-time map-view semantic segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13760–13769, 2022.
  * Zhu et al. [2020] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In International Conference on Learning Representations, 2020.