# BCSSN: Bi-direction Compact Spatial Separable Network for Collision
Avoidance in Autonomous Driving

Haichuan Li  Liguo Zhou  Alois Knoll
Chair of Robotics, Artificial Intelligence and Real-Time Systems
Technical University of Munich
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Autonomous driving has been an active area of research and development, with
various strategies being explored for decision-making in autonomous vehicles.
Rule-based systems, decision trees, Markov decision processes, and Bayesian
networks have been some of the popular methods used to tackle the complexities
of traffic conditions and avoid collisions. However, with the emergence of
deep learning, many researchers have turned towards CNN-based methods to
improve the performance of collision avoidance. Despite the promising results
achieved by some CNN-based methods, the failure to establish correlations
between sequential images often leads to more collisions. In this paper, we
propose a CNN-based method that overcomes the limitation by establishing
feature correlations between regions in sequential images using variants of
attention. Our method combines the advantages of CNN in capturing regional
features with a bi-directional LSTM to enhance the relationship between
different local areas. Additionally, we use an encoder to improve
computational efficiency. Our method takes ”Bird’s Eye View” graphs generated
from camera and LiDAR sensors as input, simulates the position (x, y) and head
offset angle (Yaw) to generate future trajectories. Experiment results
demonstrate that our proposed method outperforms existing vision-based
strategies, achieving an average of only 3.7 collisions per 1000 miles of
driving distance on the L5kit test set. This significantly improves the
success rate of collision avoidance and provides a promising solution for
autonomous driving.

##  1 Introduction

Trajectory prediction [1, 2, 3] is a critical task in autonomous driving for
collision avoidance, as it requires

(a) Front Collision

(b) Side Collision

(c) Rear Collision

Figure 1: Different collision cases during driving Figure 2: Overview of Bi-
direction Compact Spatial Separable Network

the autonomous vehicle (AV) to accurately estimate the future motion of
surrounding objects and plan its own motion accordingly shows in Fig 1. This
task is challenging due to the uncertainty of the surrounding environment and
the complexity of the motion patterns of other vehicles, pedestrians, and
cyclists. To improve the accuracy of trajectory prediction, various techniques
have been proposed. For instance, multi-modal prediction [4, 5] and
uncertainty estimation [6, 7] have been utilized to consider multiple possible
future trajectories and their probabilities. Attention mechanisms [8, 9] and
graph-based models [5, 10] have been proposed to capture the importance and
relationships among different objects. Convolutional Neural Network (CNN) has
made outstanding contributions to vision tasks and has been widely applied to
traffic scenes due to its excellent regional feature extraction capabilities.
Based on this advantage, CNN will obtain the local feature information of
sequential related frames in our network. Additionally, since the motion
trajectory is planned for AV, which means each position point has a sequential
relation (the later points depend on former points). It is necessary to
establish the relationship between each local feature of the image obtained by
CNN. To address this, some strategies use CNN plus RNN to deal with sequential
graphs as input, such as STDN [11], CRNN [12], LSTM-CNN [13], RCNN [14].

Although the above strategies have performed well in a large number of vision
tasks, their performances are still far inferior to similar-sized
convolutional neural networks counterparts, such as EfficientNets [15] and
RepVGG [16] in Fig.3. We believe this is due to the following aspects. First,
the huge differences between the sequential tasks of NLP and the image tasks
of CV are ignored. For example, when the local feature information acquired in
a two-dimensional image is compressed into one-dimensional time series
information, how to achieve accurate mapping becomes a difficult problem.
Second, it is difficult to keep the original information of inputs since after
RNN layers, we need to recover the dimension from one to three. Besides, due
to the several transformations between different dimensions, that process
becomes even harder, especially our input size is 224×224×5. Third, the
computational and memory requirement of switching between layers are extremely
heavy tasks, which also becomes a tricky point for the algorithm to run.
Higher hardware requirements as well as more running time arise when running
the attention part.

In this paper, we propose a new network structure based on CNN, Bi-LSTM,
encoder, and attention to trajectory-generating tasks in autonomous driving.
The new network structure overcomes these problems by using Bi-direction
Compress Sequential Spatial Network (BCSSN). As shown in Fig. 2, input Bird’s
Eye View (BEV) images first will be divided into three sub-graphs. Then go
through bi-directional frame-related(BIFR) blocks which consist of flatten
layer, Bi-LSTM layer, and full connect layers. After that, information will be
fed into the main stem, the convolution stem for fine-grained feature
extraction, and are then fed into a stack of SSN (Sequential Spatial Network)
blocks in Fig. 4 for further processing. The Upsampling Convolutional
Decreasing (UCD) blocks are introduced for the purpose of local information
enhancement by deep convolution, and in SSN block of features generated in the
first stage can be less loss of image resolution, which is crucial for the
subsequent trajectory adjustment task.

In addition, we adopt a staged architecture design using three convolutional
layers with different kernel sizes and steps gradually decreasing the
resolution (sequence length) and flexibly increasing the dimensionality. Such
a design helps to extract local features of different scales and, since the
first stage retains high resolution, our design can effectively reduce the
resolution of the output information in the first layer at each convolutional
layer, thus reducing the computational effort of subsequent layers. The
Reinforcement Region Unit (RRU) and the Fast MultiHead Self-Attention (FMHSA)
in the SSN block can help obtain global and local structural information
within the intermediate features and improve the normalization capability of
the network. Finally, average pooling is used to obtain better trajectory
tuning.

Extensive experiments on the lykit dataset [17] demonstrate the superiority of
our BCSSN network in terms of accuracy. In addition to image classification,
SSN block can be easily transferred to other vision tasks and serve as a
versatile backbone.

##  2 Related Works

Rule-based systems[18, 19, 20]: Rule-based systems are a type of artificial
intelligence that uses a set of rules to make decisions. These rules are
typically expressed in if-then statements and are used to guide the system’s
behavior. Rule-based systems have been used in a variety of applications,
including expert systems, decision support systems, and automated planning
systems.

Decision trees[21, 22, 23]: Decision trees are a type of machine learning
algorithm that is used for classification and regression analysis. They are
built using a tree structure, where each internal node represents a decision
based on a feature, and each leaf node represents a prediction. Decision trees
are widely used in various fields, including business, medicine, and
engineering.

Markov Decision Processes[24, 25, 26, 27] (MDPs): Markov Decision Processes
are a mathematical framework for modeling sequential decision-making problems,
where an agent interacts with an environment in discrete time steps. The
framework involves a set of states, actions, rewards, and transition
probabilities that govern the agent’s behavior. MDPs have been used in a wide
range of applications, such as robotics, finance, healthcare, and
transportation.

Bayesian networks[28, 29, 30]: A Bayesian network is a probabilistic graphical
model that represents a set of random variables and their conditional
dependencies using a directed acyclic graph. It is a powerful tool for
probabilistic inference, learning from data, and decision making under
uncertainty.

Except for the mentioned four methods, there are some popular strategies as
well. Over the past decade, autonomous driving has flourished in the wave of
deep learning, where a large number of solution strategies are based on
computer vision algorithms, using images as the primary input. The prevailing
visual neural networks are typically built on top of a basic block in which a
series of convolutional layers are stacked sequentially to capture local
information in intermediate features. However, the limited receptive field of
the small convolution kernel makes it difficult to obtain global information,
which hinders the high performance of the network on highly feature-dependent
tasks (such as trajectory prediction and planning). In view of this dilemma,
many researchers have begun to deeply study self-attention-based [31] networks
with the ability to capture long-distance information. Here, we briefly review
traditional CNN and recently proposed visual networks. Convolutional neural
network. The first standard CNN was proposed by LeCun [32] et al. and was used
for handwritten character recognition. Based on this foundation, a large
number of visual models have achieved cross-generational success in a variety
of tasks with images as the main input. Google Inception Net [33] and DenseNet
[34] showed that deep neural networks consisting of convolutional and pooling
layers can yield adequate results in recognition. ResNet [35] in Fig.3 is a
classic structure that has a better generalization ability by adding shortcut
connections to the underlying network. To alleviate the limited acceptance
domain in previous studies, some studies used the attention mechanism as an
operator for adapting patterns.

Besides, several novel visual networks have been proposed recently, which have
achieved remarkable performance in various computer vision tasks. For example,
the Mask R-CNN [36] proposed by He et al. extends the Faster R-CNN framework
with an additional mask branch to perform instance segmentation. SENet [37]
and MobileNetV3 [38] demonstrate the effectiveness of multiple paths within a
basic block. The DenseNet [34] proposed by Huang et al. introduced dense
connections between layers to improve feature reuse and alleviate the
vanishing-gradient problems. Moreover, the Transformers-based networks, such
as ViT [39] in Fig.3 proposed by Dosovitskiy et al. and DETR [40] proposed by
Carion et al., have achieved state-of-the-art performance on image
classification and object detection tasks, respectively, by leveraging the
self-attention mechanism. These novel visual networks have shown promising
results and have opened up new research directions in the field of computer
vision.

##  3 Method

(a) ResNet-50

(b) RepVGG

(c) ViT

Figure 3: Three popular network structures in vision areas. The structure of
ResNet-50 is shown in (a). The structure of RepVGG is shown in (b). The
structure of ViT is shown in (c). Figure 4: Structure of Sequential Spatial
Network (SSN)

###  3.1 Trajectory Prediction

Trajectory prediction is an essential task in autonomous driving for collision
avoidance. In this task, the goal is to predict the future trajectory of the
vehicle based on its current and past states. This prediction allows the
autonomous vehicle to plan its future path and avoid potential collisions with
other objects in the environment. What we obtain from our model and dataset
are X, Y axis position, and yaw. Since the frame slot is known, we can easily
get the velocity that AV needs. Besides, after combing fiction between ground
and wheels, wind resistance, and other physic parameters, we can calculate the
acceleration that AV needs so that we can control the motor force just like
human driving via an accelerograph. Moreover, yaw can provide a driving wheel
adjustment as well. These processes compose the basic requirements for
autonomous driving.

Some mathematical models such as kinematic or dynamic models can show these
processes. The kinematic model assumes that the vehicle moves in a straight
line with a constant velocity and acceleration. The equations of motion for
the kinematic model can be represented as:

$\begin{split}x(t)&=x_{0}+v_{0}\cos(\theta_{0})t+\frac{1}{2}a_{x}t^{2},\\\
y(t)&=y_{0}+v_{0}\sin(\theta_{0})t+\frac{1}{2}a_{y}t^{2},\\\
\theta(t)&=\theta_{0}+\omega t,\end{split}$ (1)

where $x(t)$ and $y(t)$ are the position of the vehicle at time $t$ in the $x$
and $y$ axes, respectively, $v_{0}$ is the initial velocity, $\theta_{0}$ is
the initial orientation, $a_{x}$ and $a_{y}$ are the acceleration in the $x$
and $y$ axes, respectively, and $\omega$ is the angular velocity.

On the other hand, the dynamic model takes into account the forces acting on
the vehicle, such as friction, air resistance, and gravity. Wind resistance is
an important factor to consider when predicting the trajectory of an
autonomous vehicle. The force of wind resistance can be calculated using the
following formula:

$F_{wind}=\frac{1}{2}\rho v^{2}C_{d}A,$ (2)

where $F_{wind}$ is the force of wind resistance, $\rho$ is the density of
air, $v$ is the velocity of the vehicle, $C_{d}$ is the drag coefficient, and
$A$ is the cross-sectional area of the vehicle. The drag coefficient and
cross-sectional area can be experimentally determined for a specific vehicle.

To incorporate wind resistance into the trajectory prediction model, we can
use the above formula to calculate the additional force that the vehicle must
overcome. This can then be used to adjust the predicted trajectory
accordingly.

The equations of motion for the total dynamic model can be represented as:

$\begin{split}m\frac{d^{2}x}{dt^{2}}&=F_{x}-F_{friction}-F_{wind},\\\
m\frac{d^{2}y}{dt^{2}}&=F_{y}-F_{g}-F_{friction}-F_{wind},\end{split}$ (3)

where $m$ is the mass of the vehicle, $F_{x}$ and $F_{y}$ are the forces
acting on the vehicle in the $x$ and $y$ axes, respectively, $F_{friction}$
and $F_{wind}$ are the forces due to friction and wind resistance,
respectively, and $F_{g}$ is the force due to gravity.

To predict the future trajectory of an autonomous vehicle, we need to estimate
the parameters of the kinematic or dynamic model based on the current and past
states of the vehicle. This can be done using machine learning techniques such
as regression or neural networks.

A popular neural network model for trajectory prediction in autonomous driving
is the Long Short-Term Memory (LSTM) network. LSTM networks are a type of
recurrent neural network that can capture long-term dependencies in sequential
data. The LSTM network can take as input the current and past states of the
vehicle and predict its future trajectory.

Given a sequence of input states ${s_{1},s_{2},...,s_{T}}$, where $s_{t}$
represents the state of the vehicle at time $t$, an LSTM network can predict
the future states $\hat{s}{t+1},\hat{s}{t+2},...,\hat{s}_{t+k}$, where $k$ is
the number of future time steps to predict.

The LSTM network consists of an input layer, a hidden layer, and an output
layer. The input layer takes the sequence of input states as input, and the
hidden layer contains recurrent connections that allow the network to maintain
a memory of past states. The output layer produces the predicted future
states. The LSTM network can be trained using backpropagation through time to
minimize the prediction error.

Several studies have proposed different variations of the LSTM network for
trajectory prediction in autonomous driving, such as the Social LSTM [41], the
ConvLSTM [42], and the TrajGRU [43]. These models have shown promising results
in predicting the future trajectory of the vehicle and avoiding potential
collisions with other objects in the environment.

Recent works have also explored the use of self-attention mechanisms to
capture both local and global information for trajectory prediction in
autonomous driving [44]. These models, such as the proposed SSN architecture,
combine CNN and self-attention to capture spatiotemporal features from input
data and improve the ability of sequentially related inputs.

In summary, trajectory prediction for collision avoidance in autonomous
driving is a crucial task that can be addressed using recurrent neural
networks such as LSTM or hybrid models that incorporate self-attention
mechanisms to capture both local and global information.

###  3.2 BIFR Block

To address this sequential relation problem, we also propose a bi-directional
frame-related (BIFR) block to build up relations between sequential frames, as
shown in Fig. 2. This block consists of one flatten layer, one bi-directional
LSTM, and two fully connected layers. By utilizing bi-directional LSTMs, the
information of both past and future frames can be incorporated into the
current frame, which can effectively solve the problem of unreachability and
safety of the predicted trajectory for AV.

(a) former case

(b) later case

Figure 5: Two different cases of predicted points can not reach

The trajectory planning process is based on several waypoints, and these
waypoints have a sequential relationship. The position of the later waypoints
depends on the former waypoints, and the position of the former waypoints
should consider the later waypoints as well. The potential danger of not
considering the sequential relationship between waypoints can be observed from
the examples shown in figures. The red circles represent the actual positions
of the vehicle, and the blue circles represent the predicted positions based
on the current waypoints. As can be seen from the figures shows in Fig. 5, the
former waypoints are reachable in the predicted trajectory but the latter
waypoints are not, or vice versa. Therefore, it is essential to incorporate
the sequential relationship between waypoints into the trajectory planning
process to ensure the safety and feasibility of the trajectory.

###  3.3 Network Structure Overview

Our strategy is to take advantage of CNN, encoder, Bi-LSTM, and attention by
building a hybrid network. CNN has excellent regional feature extraction
capabilities, allowing the network to accurately and quickly capture important
visual features within an image.

$\mathbf{h}i=f(\sum{j=1}^{N}\mathbf{w}j*\mathbf{x}{i-j}+b)\mathbf{h}_{i}=f(\sum{j=1}^{N}\mathbf{w}_{j}*\mathbf{x}_{i-j}+b)$
(4)

Encoder can efficiently compress and encode the input data, reducing the
computational complexity of the neural network.

$Encoder=f_{enc}(x_{1},x_{2},...,x_{T}).$ (5)

LSTM can learn and model long-term dependencies within sequential data, making
it particularly effective in modeling and predicting temporal information.

$\displaystyle f_{t}$
$\displaystyle=\sigma_{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f}),$ (6) $\displaystyle
i_{t}$ $\displaystyle=\sigma_{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i}),$
$\displaystyle\tilde{C}t$
$\displaystyle=\sigma_{c}(W_{c}x_{t}+U_{c}h{t-1}+b_{c}),$ $\displaystyle
C_{t}$ $\displaystyle=f_{t}\odot C_{t-1}+i_{t}\odot\tilde{C}t,$ $\displaystyle
o_{t}$ $\displaystyle=\sigma_{g}(W_{o}x_{t}+U_{o}h{t-1}+b_{o}),$
$\displaystyle h_{t}$ $\displaystyle=o_{t}\odot\sigma_{h}(C_{t}).$

Attention mechanisms can dynamically weight different regions of an image or
sequence, allowing the network to selectively focus on the most important
information and improving overall performance.

$\displaystyle h_{t}=\sum_{i=1}^{T_{x}}\alpha_{t,i}\cdot{h}_{i},$ (7)
$\displaystyle h_{t}=\sum_{i=1}^{T_{x}}\alpha_{t,i}\cdot{h}_{i},$
$\displaystyle\alpha_{t,i}=\frac{\exp(e_{t,i})}{\sum_{j=1}^{T_{x}}\exp(e_{t,j})},$
$\displaystyle e_{t,i}=f({s}_{t-1},{h}_{i}).$

The combination of these techniques can allow the network to effectively model
complex and dynamic relationships within a scene, allowing for more accurate
and robust predictions.

In particular, the use of attention mechanisms can help the network to better
handle variable and unpredictable inputs, allowing it to adapt to a wider
range of scenarios and improve generalization performance.

The bi-directional frame-related(BIFR) block is one of the fundamental
building blocks in our networks, especially aims to capture sequential
information in image data. The first step in the BIFR is to flatten the input
sub-graphs, which converts the input data to a one-dimensional format that can
be processed by the following layers. Next, the Bi-LSTM layer plays a central
role in capturing local sequential information. By recording the temporal
dependencies between the input data, the Bi-LSTM layer enables the network to
learn the long-term dependencies between the input and output. To enhance the
computational efficiency of the network, only the output value is kept, and
the hidden layer parameters are ignored. This allows the subsequent layers to
process the output value more efficiently. Finally, two fully connected layers
are used to map the output value produced by the Bi-LSTM layer as inputs for
later blocks, which further process the information to extract higher-level
features. Overall, the BIFR is a powerful tool for capturing sequential
information in image data and has the potential ability widely used in many
state-of-the-art computer vision networks, especially for sequentially related
inputs.

###  3.4 SSN Blcok

Resnet-50 is a powerful architecture that has proven to be effective for
various image classification tasks. Its design comprises of five stages, with
the initial stage0 consisting of a convolutional layer, a batch normalization
layer, and a maxpooling layer. The bottleneck blocks used in stages 1-4 allow
for efficient feature extraction and are followed by a fully connected layer
for classification. One of the advantages of this design is that it enables
efficient classification between different images. However, stage0 only uses
one convolutional layer with a large kernel size, which can limit the capture
of local information. To address this limitation, the main input block is
introduced, which is composed of three different kernel sizes and steps
convolutional layers. The main objective of the input block is to extract
input information quickly when the output size is large and then use the
smaller kernel sizes to capture local information as the input size decreases.
This approach helps to improve the overall performance of our model.

The proposed SSN block consists of a Reinforcement Region Unit (RRU), a Fast
Multi-Head Self-Attention (FMHSA) module and an Information Refinement Unit
(IRU), as shown in Fig. 4. We will describe these four components in the
following. After taking into account the fast processing and local information
processing of the main input block, the input information is transferred to
the subsequent blocks for subsequent processing. Furthermore, between each
block, we add a UCD layer, which consists of a convolutional layer with 1×1
kernel size and a downsampling layer which a sampling ratio is 0.5. The UCD
layer allows us to speed up the network without reducing the amount of
information in the input but maintaining the ratio between the information,
and the size of the input is reduced to half of the original size after the
UCD layer. Afterward, the feature extraction is performed by a network layer
composed of different numbers of SSN blocks, while maintaining the same
resolution of the input. Due to the existence of the self-attention mechanism,
SSN can capture the correlation of different local information features, so as
to achieve mutual dependence between local information and global information.
Finally, the results are output through an average pooling layer and a
projection layer as well as a classifier layer. The reinforcement region unit
(RRU) was proposed as a data augmentation technique for our network. Data
augmentation technique can generate high-quality augmented samples that
maintain the spatial correlation and semantic information of the original
image by selectively amplifying the informative regions of the input images.
This technique can improve the robustness and generalization of the model, and
it has been applied to various computer vision tasks such as object detection,
semantic segmentation, and image classification. Moreover, RRU can be easily
integrated with existing models and does not require additional computational
resources, making it an efficient and practical data augmentation method for
deep learning models.

In other words, a good model should maintain effective operating output for
similar but variant data as well so that the model has better input
acceptability. However, the absolute position encoding used in the common
attention was originally designed to exploit the order of the tokens, but it
breaks the input acceptability because each patch adds a unique position
encoding to it [45]. Furthermore, the concatenation between the local
information obtained by the information capture module at the beginning of the
model and the structural information inside the patch [46] is ignored. In
order to maintain input acceptability, the Reinforcement Region Unit (RRU) is
designed to extract the local information from the input to the ”SSN” module,
defined as:

$RRU(X)=Conv(Conv(X)).$ (8)

The FMHSA module in our model is designed to enhance the connection between
local information obtained from the RRU. With the use of a convolutional
layer, a linear layer, and multi-head self-attention, we can effectively
capture the correlation between different local features. This is particularly
useful for the collision avoidance task, where the trajectory prediction is
based on continuously predicted positions that are sequentially related. The
FMHSA module allows for the transfer of local information between different
areas, making it suitable for solving this problem. By improving the
connection between local information, our model is able to achieve outstanding
results in this task. Additionally, the sequential relationship between
predicted positions is taken into account, ensuring that the autonomous
driving vehicle arrives at each target position before moving on to the next
one.

The Information Refinement Unit (IRU) is used to efficiently extract the local
information obtained by FMHSA, and after processing by this unit, the
extracted local information is fed into the pooling and classifier layers. The
original FFN proposed in ViT consists of two linear layers separated by the
GELU activation [47]. First, expand the input dimension by 4 times, and then
scale down the expanded dimension:

$FFN(X)=GELU(XW1+b1)W2+b2.$ (9)

This has the advantage of using a linear function for forward propagation
before using the GELU() function, which greatly improves the efficiency of the
model operation. The proposed design concept is to address the performance
sacrifice issue of using the linear function for forward propagation in the
FFN of ViT. The solution is to use a combination of convolutional layers with
different kernel sizes and a linear function layer to achieve both operational
efficiency and model performance. First, a larger convolutional kernel is used
to capture the input information’s characteristics with a large field of view.
Then, the linear function layer is applied for fast propagation of the input
information. Finally, a smaller convolutional kernel is used to refine the
obtained information. By using this approach, the model can efficiently
process the input information while maintaining high performance during fast
propagation. This strategy is particularly useful for vision tasks such as
image classification and object detection, where the model’s performance and
operational efficiency are crucial.

$IRU(X)=Conv(L(Conv(X))),$ (10)

where L(X)=WX+b. After designing the above three unit modules, the SSN block
can be formulated as:

$\begin{split}A&=RRU(X),\\\ B&=FMHSA(A),\\\ C&=IRU(B)+B.\end{split}$ (11)

In the experiment part, we will prove the efficiency of SSN network. In
addition to its ability to capture the temporal dependencies of sequential
data, SSN also has versatility in handling different downstream tasks.
Specifically, SSN employs a self-attention mechanism to capture the
relationships between the different temporal feature maps. This mechanism can
effectively encode the dependencies between different timesteps and extract
the most relevant information for the downstream tasks. Furthermore, SSN is
designed to handle the sparsity and irregularity of the sequential data, which
is common in real-world driving scenarios. These features make SSN a promising
candidate for various applications in autonomous driving, such as trajectory
prediction, behavior planning, and decision-making.

##  4 Experiment

The autonomous driving obstacle avoidance task is crucial in ensuring safe
driving in real-life scenarios. To evaluate the effectiveness of the BCSSN
architecture, experiments were conducted using a driving map as the main
input. We use the Earlystopping function with 100 patience for finding the
best parameters’ value. Besides, we apply cosine dynamic learning rate [48]
adjustment for tuning suitable values at each iteration. To implement the
cosine dynamic learning rate adjustment, we use the following formula:

$\eta_{t}=\eta_{max}\frac{1+\cos(\frac{t\pi}{T})}{2}$ (12)

where $\eta_{max}$ is the maximum learning rate, $t$ is the current iteration,
and $T$ is the total number of iterations.

The EarlyStopping function is used to monitor the validation loss and stop the
training process when the validation loss does not improve for a certain
number of epochs. Specifically, we use the patience parameter, which
determines how many epochs to wait for the validation loss to improve before
stopping the training process. The EarlyStopping function is defined as
follows:

$\text{EarlyStopping}(V_{\text{loss}},\text{patience})$ (13)

where $V_{\text{loss}}$ is the validation loss and patience is the number of
epochs to wait for the validation loss to improve. In the experiments, the
proposed BCSSN architecture was compared with other popular models. The
experimental results in Table 1 were then analyzed to draw conclusions on the
performance of the BCSSN architecture. To assess the performance of the BCSSN
architecture, collisions were defined into three categories: front collision,
rear collision, and side collision. These collisions were caused by different
unsuitable physical parameters, and they are depicted in Fig. 1. The
experiments aimed to test the ability of the BCSSN architecture to detect and
avoid these collisions effectively. The results of the experiments were
analyzed, and the performance of the BCSSN architecture was compared with
other popular models to determine its effectiveness in addressing the obstacle
avoidance task. The analytical conclusions drawn from these experiments
provide valuable insights into the efficiency and efficacy of the proposed
BCSSN architecture.

###  4.1 Dataset and Description

The l5kit dataset is a vast collection of data, which serves as the primary
source for this study. It is an extensive dataset containing over 1,000 hours
of data collected over a four-month period by a fleet of 20 autonomous
vehicles that followed a fixed route in Palo Alto, California. The dataset
comprises 170,000 scenes, where each scene lasts for 25 seconds. Each scene
captures the perception output of the self-driving system, which provides
precise positions and motions of nearby vehicles, cyclists, and pedestrians
over time. This information is invaluable for training and testing autonomous
driving models.

In addition to the scene data, the l5kit dataset also contains a high-
definition semantic map with 15,242 labeled elements, which includes
information about the lane markings, traffic signs, and other relevant
information. This semantic map is used to help the autonomous vehicle
understand the environment and make informed decisions. Furthermore, the
dataset also includes a high-definition aerial view of the area, which
provides a top-down perspective of the environment.

Overall, the l5kit dataset is an invaluable resource for researchers and
developers working in the field of autonomous driving. With its vast
collection of data, including the perception output of the self-driving
system, semantic map, and aerial view, the dataset provides a comprehensive
understanding of the driving environment, which can be used to train and test
autonomous driving models.

###  4.2 Data Preprocessing

The l5kit dataset has a well-defined structure, consisting of three main
concepts: Scenes, Frames, and Agents. Scenes are identified by the host
vehicle that collected them and a start and end time. Each scene consists of
multiple frames, which are snapshots captured at discretized time intervals.
The frames are stored in an array, and the scene datatype stores a reference
to its corresponding frames in terms of the start and end index within the
frames array. All the frames in between these indices correspond to the scene,
including the start index, but excluding the end index.

Each frame captures all the information observed at a particular time,
including the timestamp, which the frame describes. The frame also includes
data about the ego vehicle itself, such as its position and rotation.
Additionally, the frame contains a reference to other agents (such as
vehicles, cyclists, and pedestrians) that were detected by the ego’s sensors.
The frame also includes a reference to all traffic light faces for all visible
lanes.

An agent in the l5kit dataset is an observation made by the autonomous vehicle
of another detected object. Each agent entry describes the object in terms of
its attributes, such as position and velocity, and gives the agent a tracking
number to track it over multiple frames (but only within the same scene!) and
its most probable label.

The input of this dataset is images of the Ego car, which is one of the
properties of the EgoDataset. The output of the model is the position and yaw,
which are also properties of the EgoDataset. By using this method, it is
possible to simulate vehicles’ driving as human driving actions. During the
human driving process, drivers control the accelerator and the driving wheels
to move the vehicles, where the accelerator is used for velocity and the
driving wheel for yaw. Similarly, the output of the model is also the velocity
and yaw, which can be used to simulate the trajectories of vehicles and avoid
collisions during driving.

###  4.3 Result

We conducted experiments using four different network structures and presented
their results in Table 1. We compared our model’s performance with other
transformer-based and convolution-based models and found that our model
achieved better accuracy and faster processing speed.

The results highlighted the significance of our proposed BCSSN for capturing
both local and global information, which resulted in a significant reduction
of front collision errors. Specifically, our model achieved 2.6 times fewer
front collision errors compared to RepVGG, which is 13.6 times less than
RepVGG’s error rate. Similarly, our model achieved 5.8 times fewer front
collision errors than ViT and 12.6 times less than ResNet50. Other comparisons
can observe form result table 1 directly as well. These results demonstrate
the effectiveness of the BCSSN in capturing both local and global information,
leading to significant improvements in the model’s accuracy.

BCSSN consistently outperformed other models by a large margin across all the
test scenarios. These findings indicate that the proposed BCSSN block is a
promising technique for improving the performance of autonomous driving
systems by effectively capturing both local and global information.

Table 1: Average collision times per 1000 miles. Method |  Front |  Side |  Rear |  Total
---|---|---|---|---
BC [49] |  79 |  395 |  997 |  1471
BC-perturb [49] |  14 |  73 |  678 |  765
MS Prediction [49] |  18 |  55 |  125 |  198
Urban Driver [49] |  15 |  46 |  101 |  162
ML Planner [50] |  - |  - |  - |  91.5
L5Kit (ResNet-50) [51] |  15.2 |  20.7 |  8.3 |  44.2
RepVGG [16] |  16.2 |  11.7 |  10.6 |  38.5
Vit [39] |  8.4 |  7.7 |  9.2 |  25.3
SafetyNet [50] |  - |  - |  - |  4.6
BCSSN (Ours) |  0.6 |  1.1 |  2.0 |  3.7

##  5 Feature Work

To integrate our algorithm model with the Unity virtual world, we need to
establish communication between them. We accomplish this by using Unity’s
built-in API and message-passing techniques, a first-person view, and bird eye
view figure shown in Fig. 6 . The communication protocol is established such
that the algorithm sends the input information to the virtual world, and the
virtual world returns the output information to the algorithm.

To ensure accurate physical modeling and simulation, we need to perform
coordinate transformation between the image coordinates and the world
coordinates. The process involves mapping the image coordinates to the virtual
world coordinates using the camera parameters and the intrinsic and extrinsic
calibration matrices. The resulting coordinates are then transformed to the
physical world coordinates, where the physical model can calculate the
necessary forces and physical parameters for the autonomous driving system.

(a) Bird’s eye view of virtual city

(b) First person view of virtual city

Figure 6: Autonomous driving vehicle in Unity virtual city

This transformation can be mathematically represented as:

$\begin{bmatrix}X_{world}\\\ Y_{world}\\\
Z_{world}\end{bmatrix}=R\begin{bmatrix}X_{camera}\\\ Y_{camera}\\\
Z_{camera}\end{bmatrix}+T$ (14)

where $\mathbf{R}$ is the rotation matrix, $\mathbf{T}$ is the translation
vector, and $[X_{camera},Y_{camera},Z_{camera}]$ are the coordinates in the
camera frame.

To obtain accurate values for $\mathbf{R}$ and $\mathbf{T}$, we need to
perform calibration using various methods such as checkerboard pattern or
laser scanning techniques [52]. The resulting calibration parameters can then
be used to transform the image coordinates to world coordinates for accurate
physical modeling and simulation.

##  6 Conclusion

In this paper, we introduce a novel hybrid architecture named BCSSN that is
designed to improve the ability of vision-based autonomous driving tasks as
well as other vision tasks. The BCSSN architecture combines the strengths of
several different techniques, including CNN, Bi-LSTM, encoders, and self-
attention, to capture both local and global information from sequentially
related inputs.

The CNN are used to extract local information from the input images, while the
Bi-LSTM is used to model the temporal dependency between the frames. The
encoder is used to convert the input images into a high-level feature
representation, which is then passed to the self-attention mechanism to
capture the long-range dependencies and context information.

To evaluate the effectiveness of the proposed BCSSN architecture, we conducted
extensive experiments on the Lykit dataset, which is a challenging dataset for
vision-based autonomous driving tasks. The results of these experiments
demonstrate the superiority of the proposed BCSSN architecture compared to
other state-of-the-art models. Specifically, the BCSSN architecture achieved
better performance in terms of accuracy, speed, and generalization ability.

Overall, BCSSN architecture offers a promising solution for vision-based
autonomous driving tasks and other vision tasks that require the ability to
capture both local and global information from sequentially related inputs.
Its hybrid design allows it to take advantage of the strengths of several
different techniques, resulting in improved performance and generalization
ability.

## References

  * [1] Zhenyu Wang, Zhiyong Li, Jinye Li, Wentong Cai, and Xiaoqing Liu.  Trajectory prediction for interactive virtual characters.  Multimedia Tools and Applications, 77(1):191–212, 2018.
  * [2] Kai Liu, Huan Zhang, Jingjing Li, Li Li, Liyan Li, Yu Li, Wei Song, and Chen Wang.  A hybrid model for trajectory prediction of moving objects in a multi-sensor system.  Sensors, 18(6):1753, 2018.
  * [3] Luis Rodrigues, David Goncalves, Luís Lopes, and Luís Paulo Reis.  Multiagent system for trajectory prediction of multiple football players.  In Progress in Artificial Intelligence, volume 10423, pages 375–386. Springer, 2017.
  * [4] Navneet Deo, Srikant Sahu, Mayank Vatsa, and Richa Singh.  End-to-end multi-modal multi-task learning for emotion recognition.  Expert Systems with Applications, 149:113244, 2020.
  * [5] Peng Zhao, Hanbing Yan, Yinhuan Guo, Yongqiang Cheng, and Peng Huang.  Multi-task learning with dynamic weighting for diabetic retinopathy diagnosis.  Computers in biology and medicine, 109:179–187, 2019.
  * [6] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning, pages 1050–1059. PMLR, 2016.
  * [7] Abhishek Bhattacharyya, Ujjwal Chaudhuri, and Dipti Swain.  Latent space regularized generative adversarial networks for hyperspectral image synthesis.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:1683–1696, 2020.
  * [8] Dong-Hee Kim, Youngsun Kwon, and Hyunwoo Park.  Deterministic and probabilistic deep learning approaches for prediction of traffic flow using temporal features.  IEEE Transactions on Intelligent Transportation Systems, 20(1):219–228, 2019.
  * [9] Chao Ma, Xiaojie Huang, Zhihao Yang, Yafei Wang, Chunxiao Yang, Dengshi Huang, Zhiheng Wang, Wayne Wang, Tobias Weyand, Wei Sun, et al.  Trafficpredict: Trajectory prediction for heterogeneous traffic-agents.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1307–1317, 2019.
  * [10] Changqing Yu, Yabo Dong, Jiangnan Pang, Xiaohu Huang, Xiaowei Zhang, and Jian Sun.  Grnet: Gridding residual network for dense point cloud completion.  arXiv preprint arXiv:2103.16287, 2021.
  * [11] Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, Yanwei Yu, and Zhenhui Li.  Modeling spatial-temporal dynamics for traffic prediction.  arXiv preprint arXiv:1803.01254, 1:9, 2018.
  * [12] Andrew Shin, Seunghyun Lee, Hyun-Woo Lee, and Hyunwoong Kim.  Deep convolutional neural networks and recurrent neural networks for detection of fetal hypoxia from cardiotocography.  IEEE Journal of Biomedical and Health Informatics, 21(1):274–282, 2017.
  * [13] Nicolas Ballas, Li Yao, Christopher Pal, and Aaron Courville.  Delving deeper into convolutional networks for learning video representations.  arXiv preprint arXiv:1511.06432, 2015.
  * [14] Liang-Chieh Chen and Yi Yang.  Recurrent convolutional neural network for object recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3367–3375, 2017.
  * [15] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [16] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun.  Repvgg: Making vgg-style convnets great again.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13733–13742, 2021.
  * [17] John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska.  One thousand and one hours: Self-driving motion prediction dataset.  In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 409–418. PMLR, 16–18 Nov 2021\.
  * [18] Richard W Harper, Michael W Lehman, James E Miller, Kevin B Johnson, James L Myers, Thomas R Martin, Thomas A Platts-Mills, Peter J Sterk, Vito G Brusasco, et al.  Rule-based expert systems for diagnosis and management of asthma.  In AMIA Annual Symposium Proceedings, volume 1993, page 370. American Medical Informatics Association, 1993.
  * [19] Ihsan Yilmaz.  Rules versus decision trees for classification of medical data.  Journal of medical systems, 28(2):125–130, 2004.
  * [20] Franziska Hinkelmann, Michael Brandon, Bonnie Guang, and Hans V Westerhoff.  Rule-based modeling and simulation of biochemical systems with rulemonkey.  In International Workshop on Computational Methods in Systems Biology, pages 182–191. Springer, 2004.
  * [21] Leo Breiman, Jerome H Friedman, Richard A Olshen, and Charles J Stone.  Classification and regression trees.  Wadsworth International Group, 1984.
  * [22] Bhavani Mehta and Rakesh Agrawal.  Using decision trees to improve case-based learning.  In Proceedings of the Thirteenth International Conference on Machine Learning, pages 327–335. Morgan Kaufmann Publishers Inc., 1996.
  * [23] Konstantinos Veropoulos, Colin Campbell, and Nello Cristianini.  Controlling the complexity of decision trees by pruning and visualization.  Proceedings of the British Machine Vision Conference, 1:129–138, 1999.
  * [24] Martin L Puterman.  Markov decision processes: discrete stochastic dynamic programming.  John Wiley & Sons, 2014.
  * [25] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937. PMLR, 2016.
  * [26] Stéphane Ross, Geoffrey J Gordon, and J Andrew Bagnell.  Reinforcement and imitation learning via interactive no-regret learning.  Foundations and Trends® in Machine Learning, 7(1):1–129, 2014.
  * [27] Oliver Lemon and Shimon Whiteson.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  arXiv preprint arXiv:1904.00943, 2019.
  * [28] Daphne Koller and Nir Friedman.  Probabilistic graphical models: principles and techniques.  MIT press, 2009.
  * [29] Finn V Jensen.  Bayesian networks and decision graphs.  In Springer, pages 17–39. Springer, 2001.
  * [30] David Heckerman.  Learning bayesian networks: The combination of knowledge and statistical data.  Machine learning, 20(3):197–243, 1995.
  * [31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [32] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural computation, 1(4):541–551, 1989.
  * [33] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [34] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [36] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [37] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [38] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * [39] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [40] Nicolas Carion, Francisco Massa, Alexander Kirillov, and Ross Girshick.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [41] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese.  Social lstm: Human trajectory prediction in crowded spaces.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–971, 2016.
  * [42] SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Ki Wong, and Wing-Kin Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  arXiv preprint arXiv:1506.04214, 2015.
  * [43] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In European Conference on Computer Vision (ECCV), 2018.
  * [44] Yingfeng Li, Zeyu Qin, Wenyu Sun, Guanbin Li, and Liang Wang.  A multi-head self-attention network for autonomous driving trajectory prediction.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 12271–12277. IEEE, 2021.
  * [45] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen.  Conditional positional encodings for vision transformers.  arXiv preprint arXiv:2102.10882, 2021.
  * [46] Md Amirul Islam, Sen Jia, and Neil DB Bruce.  How much position information do convolutional neural networks encode?  arXiv preprint arXiv:2001.08248, 2020.
  * [47] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  arXiv preprint arXiv:1606.08415, 2016.
  * [48] Leslie N Smith.  A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.  arXiv preprint arXiv:1803.09820, 2018.
  * [49] Oliver Scheel, Luca Bergamini, Maciej Wolczyk, Błażej Osiński, and Peter Ondruska.  Urban driver: Learning to drive from real-world demonstrations using policy gradients.  In Conference on Robot Learning, pages 718–728. PMLR, 2022.
  * [50] Matt Vitelli, Yan Chang, Yawei Ye, Ana Ferreira, Maciej Wołczyk, Błażej Osiński, Moritz Niendorf, Hugo Grimmett, Qiangui Huang, Ashesh Jain, et al.  Safetynet: Safe planning for real-world self-driving vehicles using machine-learned policies.  In 2022 International Conference on Robotics and Automation (ICRA), pages 897–904. IEEE, 2022.
  * [51] John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska.  One thousand and one hours: Self-driving motion prediction dataset.  In Conference on Robot Learning, pages 409–418. PMLR, 2021.
  * [52] Jia Zhang and Sanjiv Singh.  An evaluation of camera calibration methods for autonomous driving.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2344–2353, 2018.