# PIP-Net: Pedestrian Intention Prediction in the Wild

Mohsen Azarmi 1, and Mahdi Rezaei 1, and He Wang 2, and Sebastien Glaser 3
Corresponding author: M. Rezaei (email<EMAIL_ADDRESS> 1Institute for
Transport Studies, University of Leeds, United Kingdom
2Department of Computer Science, University College London, United Kingdom
3Centre for Accident Research and Road Safety, Queensland University of
Technology, Australia

###### Abstract

Accurate pedestrian intention prediction (PIP) by Autonomous Vehicles (AVs) is
one of the current research challenges in this field. In this article, we
introduce PIP-Net, a novel framework designed to predict pedestrian crossing
intentions by AVs in real-world urban scenarios. We offer two variants of PIP-
Net designed for different camera mounts and setups. Leveraging both kinematic
data and spatial features from the driving scene, the proposed model employs a
recurrent and temporal attention-based solution, outperforming state-of-the-
art performance. To enhance the visual representation of road users and their
proximity to the ego vehicle, we introduce a categorical depth feature map,
combined with a local motion flow feature, providing rich insights into the
scene dynamics. Additionally, we explore the impact of expanding the camera’s
field of view, from one to three cameras surrounding the ego vehicle, leading
to enhancement in the model’s contextual perception. Depending on the traffic
scenario and road environment, the model excels in predicting pedestrian
crossing intentions up to 4 seconds in advance which is a breakthrough in
current research studies in pedestrian intention prediction. Finally, for the
first time, we present the Urban-PIP dataset, a customised pedestrian
intention prediction dataset, with multi-camera annotations in real-world
automated driving scenarios.

###### Index Terms:

Autonomous vehicles, pedestrian crossing behaviour, pedestrian intention
prediction, computer vision, deep neural networks.

##  I Introduction

Pedestrians are the most vulnerable road users and face a high risk of fatal
accidents [1]. Ensuring pedestrian safety in automated driving, particularly
in mixed AV-pedestrian traffic scenarios, heavily relies on the AV’s
capability in “pedestrian intention prediction (PIP)” [2]. A PIP system
determines if a pedestrian is likely to cross the road shortly (within the
next few seconds). This study aims to investigate the critical visual clues
that pedestrians exhibit when they intend to cross the road, and then provide
a model which predicts crossing behaviour a few seconds in advance.

Anticipating pedestrian crossing behaviour is a difficult task due to various
environmental factors that affect human intention [3, 4]. Even in the
simulated scenarios in which the majority of parameters are under control,
crossing prediction is a challenging endeavour [5]. Factors like interactions
with other pedestrians, traffic signs, road congestion, and vehicle speed can
influence pedestrians’ tendency to cross the road in front of AVs [6].

Computer vision plays a crucial role in enabling AVs to perceive their
surrounding environment by analysing the visual data captured via multiple
sensors, such as cameras, LiDAR, Radar, etc. Learning-based models, in
particular deep neural networks (DNNs), have shown remarkable success in
various computer vision tasks, including scene understanding, semantic
segmentation [7], road users classification, localisation [8], and motion
prediction [9]. Figure 1 illustrates some of the perceivable factors such as
depth, pedestrian pose, and surrounding objects that an autonomous vehicle
should consider to interpret the scene and estimate the pedestrians’
intention. DNNs are particularly effective at learning complex patterns and
features from visual data, making them a natural fit for tasks that involve
analysing images or videos to comprehend pedestrian behaviour [10, 11, 12].
They also offer significant capabilities in multi-modal integration by
providing a neural-based mechanism to process and fuse all the perceived
information from diverse sensors. This integration may enhance the overall
understanding of the environment and help to make more accurate and safer
decisions [13].

Figure 1: Pedestrians’ crossing intention prediction in complex urban
scenarios via contextual data analysis and a multi-camera perception setup.

Several datasets, such as JAAD [14], PIE [15], and STIP [16], use onboard
camera recordings and their data are publicly released for the study of
pedestrians’ behaviour before and during road crossing. However, most of the
current research works suffer from supplying a multi-camera in-cabin setup to
leverage the benefits of sensor fusion and multi-modal perception. In addition
to the above-mentioned datasets, some baseline approaches [17] have been also
established for analysing the visual cues and signals that pedestrians emit
through their body language and positioning. The approaches highlight the
benefits of combining these features with contextual information. Contextual
information may include factors such as the road’s location, the time of day,
weather conditions, the presence of traffic signals or crosswalks, the type of
road (urban, suburban, rural), and the position and behaviour of other
vehicles near the scene [18]. To the best of our knowledge, no extensive
research has been conducted to understand and interpret such contextual
details and their effects on pedestrian’s decision-making.

In this study, we propose a customised DNN-based framework, called “PIP-Net”
that takes various features of pedestrians, the environment, and the ego-
vehicle state into account, to learn the context of a crossing scenario and
consequently predict the intention of pedestrians in real-world AV urban
driving scenarios. The main contributions of this research are highlighted as
follows:

  * •

A novel feature fusion model is presented to integrate AV’s surrounding
cameras and combine visual and non-visual modalities, as well as a hybrid
feature map that incorporates depth and instance semantic information of each
road user to comprehend the latent dynamics in the scene.

  * •

Introducing the multi-camera “Urban-PIP dataset”, which includes various real-
world scenarios of pedestrian crossing for autonomous driving in urban areas.

  * •

We examine the effectiveness of the various input features, temporal
prediction expansion, and the worthiness of expanding the vehicle field of
view from one camera to three cameras based on the latest Waymo car camera
setup [19] to ensure the developed model will be in line with the current
technology developments in the AV industry.

  * •

Finally, we evaluate the effectiveness of the proposed model on the widely
utilised PIE dataset and the introduced Urban-PIP dataset, outperforming the
state-of-the-art (SOTA) for predicting pedestrian actions [11].

##  II Related Works

Recently, pedestrian crossing intention prediction research has surged and
gained significant attention within the autonomous driving research community
[20, 10, 11, 12]. Most current methods mainly address the problem by taking
two aspects into account: Discovering influential factors and features for
interpreting road users’ interactions [5, 6, 4], and designing the analytic
model to predict the pedestrians’ crossing intention [18, 14, 15, 16, 17].
Both research directions mainly utilise advanced learning-based techniques.
Deep learning methods have been fostered on multiple features of pedestrians
and the environment, whether derived from annotations, visual information from
videos or their combinations [2]. The following two subsections are dedicated
to introducing approaches that utilise DNN-based architectures for spatio-
temporal analysis and feature selection/fusion.

###  II-A Spatio-temporal Analysis

Recently, there has been a shift from still image analysis to the
incorporation of temporal information into the prediction models. Rather than
relying on individual images, most contemporary methods utilise sequences of
input images for decision-making by their prediction models. This adaptation
recognises the significance of temporal data in enhancing the prediction task,
resulting in what is known as spatio-temporal modelling.

Spatio-temporal modelling can be achieved through a two-step process.
Initially, visual (spatial) features per frame can be extracted using 2D
convolutional neural networks (CNNs) [21] or graph convolution networks (GCNs)
[16]. Subsequently, these extracted features are then fed into RNNs, such as
the long short-term memory (LSTM) [15] or the gated recurrent unit (GRU)
models [22, 23, 24]. For instance, in [25, 26], 2D convolutions are employed
to extract visual features from image sequences, while RNNs encode the
temporal relationships among these features. These sequentially encoded visual
features are then inputted into a fully-connected layer to generate the
ultimate intention prediction.

An alternative approach to extracting sequential visual features involves the
utilisation of 3D CNNs (Conv3D) [27]. This technique directly captures spatio-
temporal features by substituting the 2D kernels within the convolution and
pooling layers of a 2D CNN with their 3D equivalents. For instance, in works
such as [28] and [29], a framework based on a 3D CNN, specifically a 3D
DenseNet, is employed to directly extract sequential visual features from
sequences of pedestrian images. The ultimate prediction is then made using a
fully-connected layer. Transformer architecture has also been utilised in
another study [30] to tokenise the temporal input features and subsequently
judge the pedestrians’ intention.

###  II-B Feature Selection and Integration

Instead of pursuing an end-to-end approach for modelling visual features, it
is possible to treat various types of information such as the pedestrian’s
bounding box, body pose keypoints, vehicle movement, and the broader
contextual backdrop as distinct input channels for the prediction model [12].
This necessitates the development of a fusion approach for amalgamating this
diverse information.

The investigation into the types of features, such as pedestrians and
environmental context, is still ongoing. For instance, pedestrian-to-vehicle
distance is considered one of the most influential factors in pedestrians’
decision to cross [5]. This feature is typically estimated as a single measure
between the target pedestrian and the ego vehicle [4]. Alternatively, a depth
map of the scene (see Figure 2e) can be used to assess the distance from other
road users and possibly reveal the underlying dynamics [31]. However, the
depth map is susceptible to noise due to rough estimation, which can lead to
inaccuracies in multiple pedestrian crossing scenarios [32].

Studies such as [33, 34, 35] have incorporated human poses or skeletons into
pedestrian crossing prediction tasks, using pose keypoints extracted from
pedestrian images to construct classifiers. This approach has shown improved
prediction accuracy, but often neglects other important features or lacks
attention to feature integration.

On the other hand, some studies specifically concentrated on feature
integration. For instance, vision and non-vision branches fusion [36] suggest
how to efficiently combine diverse data modalities at different stages of a
DNN model to surge the intention prediction accuracy. Another study [10] is
conducted to merge two visual and three non-visual elements of the pedestrian,
scene, and subject vehicle in a multi-stream network. From a different
perspective, in studies such as [11, 37], local and global contextual
information has been weighted by an attention mechanism [38] and fused
together to apply a prediction on Joint Attention in Autonomous Driving (JAAD)
[14] and Pedestrian Intention Estimation (PIE) [15] datasets.

Figure 2:  Analysing pedestrians’ features and traffic scene dynamics through:
a) Pedestrian localisation, highlighted with bounding boxes, b) Gesture
understanding with pose estimation, c) Object categorisation through
segmentation, d) Understanding global motion patterns from optical flow, and
e) Estimating distance via global depth heatmap.

###  II-C Research Gaps

Pedestrian crossing intention highly relies on the distance of the AV to the
pedestrian and the relative distance of the pedestrian to other road users
which may fall into various categories of instance segmentation (e.g., cars,
other pedestrians, etc.). None of the reviewed research has considered the
simultaneous impact of both features on pedestrian intention.

On the other hand, to the best of our knowledge, no prior study has considered
instance segmentation to smooth and normalise the distance measurement of road
user instances.

In this article, we propose a new concept of Categorical Depth which
integrates the classic noisy depth measurement with instance segmentation to
gain more accurate depth information.

As another issue, the reviewed models, often have limited generalisability and
are incapable of performing in the wild and real-world scenarios, as they have
not been tested under authentic autonomous driving conditions [20]. Our study
focuses on the real-world Waymo dataset which is collected from an AV’s field
of view.

Lastly, there is a shortage of dedicated neural network architectures capable
of effectively accommodating and extracting maximal multi-camera information
from around the AV for context recognition, hence an accurate model for
predicting pedestrian crossing intentions. Camera integration is proposed in
this study to cope with the limited field of view.

##  III Methodology

We propose the PIP-Net prediction model, which is based on deep neural
networks for predicting pedestrian crossing intention. The model incorporates
spatial-temporal features such as road users’ positioning, pose, and dynamic
movements, along with a hybrid feature map that includes categorised semantic
and depth information as input to the network. A multi-camera stitching and
integration model is developed to facilitate panoramic viewing, enabling
synchronised pedestrian ID assignment and tracking across the entire multi-
view scene, thus enhancing the PIP-Net model’s understanding of spatial
characteristics and contextual information.

An overview of the proposed architecture is illustrated in Figure 3. The input
features are categorised into spatial kinematic data and contextual data, and
they are passed to the model through distinct pipelines based on their data
types. Finally, we employ recurrent and attention modules to facilitate the
processing of temporal data.

Figure 3: The overview of the proposed DNN-based framework. PIP-Net receives
Spatial Kinematic and Spatial Context data, and applies multi-modal feature
fusion and multi-camera integration with temporal and attention-based analysis
to predict pedestrians’ crossing intentions. Shift and Padding units adjust
the location of the target pedestrian with respect to the corresponding
camera. Aggregation module (A) fuses the cameras’ features as shown in Figure
5.

###  III-A Spatial Kinematics

Kinematic input data includes the positioning of the pedestrian in the scene
with reference to the detected pedestrian bonding box $P_{bb}$, pedestrian
body pose keypoints $P_{bp}$, and the ego-vehicle speed $V_{s}$.

The data is arranged in a gated recurrent unit (GRU) layer [22], beginning
with the Bounding Box feature $P_{bb}$. It indicates the location of the
pedestrians which is detected through the customised You-Only-Look-Once
algorithm for road user detection [39]. This feature is defined as:

$P_{bb}=\\{b^{t-m}_{i},b^{t-m+1}_{i},...,b^{t}_{i}\\},$ (1)

where $b_{i}=[x_{1},y_{1},x_{2},y_{2}]\in\mathbb{R}^{4}$ represents the
coordinates of a pedestrian bounding box. It consists of the top-left
$([x1,y1])$ and the bottom-right coordinates $([x2,y2])$. The dimension of the
bounding box matrix $P_{bb}$ is determined as $m\times 4$, where $m$ is the
observation time which indicates the number of frames that are observed to
predict the pedestrian intention. We define $t$ as the decisive moment, 0.5
$\thicksim$ 4 seconds before the crossing event.

The Body Pose feature $P_{bp}$ is defined as:

$P_{bp}=\\{p^{t-m}_{i},p^{t-m+1}_{i},...,p^{t}_{i}\\},$ (2)

where the pose keypoints are obtained using YOLO-Pose [40], which estimates
the pose of a person by detecting 17 keypoints joints, including the
shoulders, elbows, wrists, hips, knees, ankles, eyes, ears, and nose. The
keypoints are represented by a 34-dimensional vector, $p_{i}$, which contains
the 2D coordinates of each joint for the $i$-th pedestrian at time $t$.

The Vehicle Speed is also defined as:

$V_{s}=\\{s^{t-m}_{i},s^{t-m+1}_{i},...,s^{t}_{i}\\},$ (3)

where $s_{i}$ refers to the exact speed of the ego-vehicle in km.

###  III-B Spatial Context

Contextual input data includes pedestrian features, such as a pedestrian-
bounded image (Local Content, $P_{lc}$) and corresponding motion flow analysis
of the pedestrian (Local Motion, $P_{lm}$), the environment features like
semantic segmentation of the scene (Semantic Context, $E_{sc}$), as well as
our proposed hybrid feature map (Categorical Depth, $E_{cd}$), which refines
depth information for specific pedestrians and vehicles in the scene. The
mentioned features are obtained by using an ImageNet pre-trained VGG19 network
as the backbone CNN, with a maximum pooling layer as suggested in [36].
Subsequently, a GRU is applied recursively to process each of these features.

The Local Content feature $P_{lc}$ is defined as:

$P_{lc}=\\{lc^{t-m}_{i},lc^{t-m+1}_{i},...,lc^{t}_{i}\\},$ (4)

where $lc_{i}$ denotes the feature vector that is output by applying the CNN
backbone to an RGB image. The image contains an individual pedestrian, cropped
based on the bounding box location and subsequently warped to dimensions of
$224\times 224\times 3$ pixels, which is the optimum spatial size in the
network [30].

The resulting input feature vector is extracted as $(m,512)$ via a Conv3D
layer. It is then passed through a 3D max-pooling layer (MP3D) with a kernel
size of $4\times 4$ and a GRU module. This process yields a $(m,128)$ vector,
where $m$ represents the observation time.

The Local Motion feature $P_{lm}$ is derived from the dense optical flow
analysis within the pedestrian-bounded image. This analysis is more consistent
than examining the entire scene, which can be affected by ego-vehicle motions.
We opt for a more advanced optical flow approach using Flownet2 [41]. This
deep learning-based method offers improved accuracy and faster run-time
performance. The Local Motion is defined as:

$P_{lm}=\\{lm^{t-m}_{i},lm^{t-m+1}_{i},...,lm^{t}_{i}\\},$ (5)

where $lm_{i}$ is considered as the localised $i$-th pedestrian motion
descriptor. A CNN layer is used to extract a feature vector of size $(m,256)$.
This vector is then inputted into a GRU layer, resulting in an $(m,128)$
vector, suitable for concatenation with the Local Content feature vector.

The Semantic Context feature $E_{sc}$ is defined as:

$E_{sc}=\\{sc^{t-m},sc^{t-m+1},...,sc^{t}\\},$ (6)

where $sc$ refers to the semantic segmentation of objects within the entire
scene encompassing road structure and users. This feature ensures that the
model considers the spatial distribution of classes for both moving and static
objects within the scene. The semantic information is extracted by Slot-VPS
[42] model, which is a panoptic video segmentation algorithm that not only
offers semantic segments but also a unique ID for each instance of the objects
in the scene. The segmented classes include 8 dynamic classes (person, rider,
car, truck, bus, train, motorcycle, and bicycle) and 11 static classes
(traffic light, fire hydrant, stop sign, parking meter, bench, handbag, road,
sidewalk, sky, building, and vegetation).

The Categorical Depth feature $E_{cd}$ is defined as:

$E_{cd}=\\{cd^{t-m},cd^{t-m+1},...,cd^{t}\\},$ (7)

where $cd$ represents the hybrid feature map showing the spatial distribution
and distance of various instances of pedestrians and vehicles within the
scene. The depth data are initially estimated using the ManyDepth model [43]
and encoded in a heatmap representation, resulting in a global depth heatmap.
As illustrated in Figure 2e, high-intensity spots (white and oranges) indicate
proximity to the ego vehicle, while low-intensity spots (navy blue and black)
represent greater distances. However, our experiments revealed that the global
depth heatmap is unreliable due to inconsistencies in providing clear object
boundaries. To address this, the pedestrian and vehicle instances are cropped
using instance masks obtained from the Slot-VPS, as shown in Figure 4a.
Subsequently, the intensity of pixels within each instance is normalised by
averaging, yielding a normalised heatmap as seen in Figure 4b. This process
ensures a clear and consistent depth estimation for each instance of classes,
as depicted in Figure 4c.

Both inputs, $E_{sc}$ and $E_{cd}$, undergo extraction via a Conv3D layer to
be assessed for the spatio-temporal analysis. The feature dimensions are
gradually reduced $(512\rightarrow 256\rightarrow...\rightarrow 64)$ by
repeatedly subjecting them to max-pooling layers. This process not only
selects the most important information from the local neighbourhood of each
pooling window but also reduces the spatial dimensions (width and height) of
the feature maps and the computational complexity of the network. Then the
features are organised using a flattened layer, resulting in a one-dimensional
array that is suitable for concatenation and can be fed into a fully-connected
layer (FC). Finally, the data is passed through a GRU module. The outputs of
the three GRUs are combined and concatenated into a single output, which is
then passed through an attention mechanism.

(a) The depth heatmap of instances

(b) The normalised heatmap per each instance

(c) Categorical depth map

Figure 4: Colour-coded visualisation of road users’ distance to the ego-
vehicle after the tripe camera stitcheing. 4a is the proportion of the global
depth heatmap which pedestrians and vehicles occupy; 4b is the normalised
heatmap values per each instance; 4c is a visualised $E_{cd}$ feature in an
RGB format in which pedestrians and vehicles integrated into the blue and red
channel, respectively, and high-intensity values indicate closer users. A
categorical depth map has a positive effect on removing the camera stitching
effect and increasing the clarity and saliency of pedestrians and vehicles in
the scene.

###  III-C Cameras Features Integration

The incorporation of multiple cameras might be beneficial for capturing
complex traffic scenarios, such as intersections, thanks to providing a
surrounding field of view. In these scenarios, pedestrians may approach the
road from the sides rather than directly in front of the vehicle. They may
also choose to cross the road while a vehicle is changing lanes or making a
turn. By incorporating left and right-side cameras, we can gather critical
information about pedestrians in adjacent lanes or at the side of the vehicle.
The Waymo dataset is one of the best options with three cameras ($c=3$) that
also offer a diversity of real-world pedestrian crossing scenarios. The
cameras are named front-left (FL), front (F), and front-right (FR) positioned
from the AV’s left to right angles (as shown in Figure 2). The synchronised
videos provided have an approximately 11% overlap along the edges. These
overlapping areas can introduce redundancy in data and make it challenging to
precisely determine the pedestrian’s position, movement, and intention when it
moves from one camera scene to another. Therefore, we merge the cameras using
the Panoptic stitching over time approach [19], excluding the overlapping
regions and giving higher priority to the front-view camera. Figure 2
illustrates an example of different types of features that have been extracted
from three cameras, and then stitched together to constitute a single wide
image.

Figure 5: The multi-camera feature aggregation module. In this example, the
front-left camera (FL) is the sentinel camera, where the target pedestrian is
observed. The padding module expands the Camera Index value and provides a
camera indicator mask for combining with Sense Context and Categorical Depth
features.

We define Sentinel Camera as a variable that indicates the index of the camera
($C_{ix}$) on which the target pedestrian has been observed. Using the Camera
Index, we can adjust the pose and bounding box coordinates with respect to the
sentinel camera. This task will be accomplished by the Shift unit, as shown in
Figure 3 in cyan, which extends the global coordinates from the leftmost
camera to the rightmost one, and applies these adjustments to the inputs. In
this context, the Padding unit is responsible for generating a zero binary
mask of size $c\times 512\times 512$, where the $c$-th dimension corresponds
to the sentinel camera being set to one. Here, $c$ represents the number of
cameras. As the aggregation module (A) shown in Figure 5, the binary masks are
combined with the feature vectors generated by the VGG19 network for each
$E_{sc}$ and $E_{cd}$. Subsequently, a pointwise convolution (Conv $1\times
1$) operator aggregates all the features across cameras. This process combines
features from different channels (cameras) at each spatial location, allowing
a weighted combination of input features.

###  III-D Temporal and Attention Module

To account for the temporal context of input features, GRU is employed. When
describing the recursion for the GRU equation, the variables at the $j$-th
level of the stack can be outlined as follows:

$\displaystyle
z^{t}_{j}=\sigma(x^{t}_{j}W^{xz}_{j}+h^{t}_{j}W^{hz}_{j}+b^{z}_{j}),$ (8)
$\displaystyle
r^{t}_{j}=\sigma(x^{t}_{j}W^{xr}_{j}+h^{t}_{j}W^{hr}_{j}+b^{r}_{j}),$ (9)
$\displaystyle\tilde{h}^{t}_{j}=tanh(x^{t}_{j}W^{x}_{j}+(r^{t}_{j}\odot
r^{t-1}_{j})W^{h}_{j}+b),$ (10) $\displaystyle h^{t}_{j}=(1-z^{t}_{j})\odot
h^{t-1}_{j}+z^{t}_{j}\odot\tilde{h}^{t}_{j},$ (11)

where $\sigma(.)$ denotes the logistic sigmoid function $x^{t}_{j}$ is the
input feature at time step $t$. The reset and update gates at time step $t$
are denoted as $r^{t}_{j}$ and $z^{t}_{j}$, respectively, and the weights
between the two units are represented by $W$. The hidden state at the previous
time step and the current time step are represented by $h^{t-1}_{j}$ and
$h^{t}_{j}$, respectively.

To assess the significance of the processed features during network training,
the attention mechanism [38]) is utilised to focus on specific segments of
features, thereby enhancing the effectiveness of feature analysis. The
resulting vector from the attention module is defined as follows:

$\displaystyle A=tanh(W_{c}[h_{c}:h_{m}]),$ (12) $\displaystyle
h_{c}=\sum_{s^{t}}\alpha_{t}h_{s^{t}},$ (13)

where $W_{c}$ represents a weight matrix, $h_{c}$ denotes the cumulative sum
of all attention-weighted hidden states, $h_{m}$ signifies the final hidden
state of the encoder, $h_{s^{t}}$ corresponds to the preceding hidden state of
the encoder, and $\alpha^{t}$ denotes the vector of attention weights, which
is defined as follows:

$\displaystyle\alpha_{t}=\frac{\mbox{exp}(score(h_{m},\tilde{h}_{s})}{\Sigma^{t_{m}}_{s^{t}=1}\mbox{exp}(score(h_{m},\tilde{h}_{s^{t}}))},$
(14) $\displaystyle score(h_{m},h_{s^{t}})=h^{T}_{m}W_{p}h_{s^{t}}$ (15)

where $t_{m}$ is the input sequence length at time $t$. $h^{T}_{m}$ represents
the transpose of the $h_{m}$ vector, and $W_{p}$ is a weight matrix that can
be estimated through the training phase of the network.

In the tile of the network, the outputs of the attention modules are
concatenated and then forwarded through the last attention module and an FC
layer. The ultimate output, normalised to a range between zero and one using
the Softmax function, represents the prediction for pedestrian crossing
intention.

##  IV Experiments

In this section, we conduct four distinct experiments to thoroughly assess the
robustness of the proposed framework. Each experiment is designed to provide
unique insights into various aspects of the model’s performance. We compare
our model against the PIE dataset in four different intervals ranging from 1
to 4 seconds, allowing us to scrutinise the model’s predictive capabilities
for an in-time response in different driving scenarios. The second experiment
examines the impact of the introduced Categorical Depth and Local Motion
features in enhancing the framework’s prediction accuracy. The third
experiment evaluates the model’s generalisability and reliability on a diverse
dataset from Waymo’s self-driving vehicles. This investigation ensures that
the framework performs effectively across different real-world scenarios.
Lastly, we investigate the scalability of the model by assessing the
framework’s ability to handle one to three cameras simultaneously, expanding
its view angles. This provides insights into the model’s efficiency when
processing information from multiple cameras. This multi-faceted evaluation
allows us to gain a detailed understanding of the model’s effectiveness, in
comparison to a singular experiment.

###  IV-A Datasets

The JAAD [14] and STIP [16] datasets suffer from no annotation in terms of
ego-vehicle speed values. Furthermore, there is a slight bias in these
datasets, as the majority of annotations indicate the cases of crossing which
may not result in effective training of deep models. Therefore, the
evaluations were conducted on the Pedestrian Intention Estimation (PIE) [15]
dataset, which is extensively employed in the majority of prior studies.
Additionally, we utilised our custom dataset named Urban-PIP, specifically
annotated for pedestrian crossing behaviour, built upon the Waymo [19]
dataset. Waymo is a widely used dataset for traffic perception by AVs thanks
to the diversity of the video data from urban and rural environments under
various driving conditions and situations. The specifications of datasets are
briefly mentioned in Table I.

TABLE I: Dataset Specifications
---
Specification | PIE | Urban-PIP
Autonomous Driving | No | Yes
Number of Cameras | 1 | 3
Auxiliary Sensors | OBD | LiDAR, Radar, IMU
Video clip lengths | 10 min | 16 sec
Total Number of Frames | 909,000 | 32,790
Total Number of Annotated Frames | 293,000 | 32,790
Total Number of Pedestrians | 1,842 | 1,481
Crossed Pedestrians | 512 | 409
Not Crossed Pedestrians | 1,328 | 1,072

####  IV-A1 PIE dataset

The dataset is recorded on a sunny clear day for 6 hours in HD format
($1920\times 1080$). Each video segment lasts approximately 10 minutes,
resulting in a total of 6 sets. We utilised approximately 50% (880 samples) of
the dataset for training, 40% (719) for testing, and 10% (243) for validation
as per the same split proportion as [11]. Regarding occlusion levels, partial
occlusion is defined when an object is obstructed between 25% and 75%, while
full occlusion occurs when the object is obstructed by 75% or more. The
dataset includes the vehicle speed, heading direction, and GPS coordinates.

####  IV-A2 Urban-PIP dataset

The dataset is recorded under various weather and daytime conditions in three
geographical locations using a multi-sensor setup. This multi-modal dataset is
collected via a combination of LiDAR, cameras, radar, and IMU sensors mounted
on the ego-vehicle. LiDAR provides 360∘ field of view with approximately a
300-meter range by beaming out millions of laser pulses per second and
measuring the time of laser beam flight from the sensor to the surface of an
object, then reflecting from the object to the sensor on the ego-vehicle. The
radar system has a continuous 360∘ view to track the presence and speed of
road users in front, behind and sides of the vehicle. The front cameras (FL,
F, and FR) simultaneously capture the traffic scene videos in HD format
($1920\times 1080$). The IMU module uses accelerometers and gyroscopes with
input from GPS, maps, wheel speeds, as well as laser and radar measurements to
provide position, velocity, and heading information to the vehicle.

In this study, the experiments are conducted using camera sensors as they
provide rich visual information, including detailed information about
pedestrian behaviour, body language, and contextual information that can be
crucial for predicting crossing intentions. Also, the affordability of camera
sensors has made them a practical choice for current research on intention
prediction. We annotated 1,481 pedestrian crossing intentions from the front
cameras including 448 in the front-left camera, 541 from the front camera, and
492 from the front-right camera.

To assess models limited to a single camera, we introduce a subset, named
Frontal-Urban-PIP, focusing on pedestrians observed only by the front camera.
This subset, featuring 55 pedestrians with crossing intentions and 129
without, ensures a fair comparison will be conducted with similar methods
which are limited to a single camera only.

###  IV-B Implementation Settings

The proposed model was executed on a CUDA parallel computing platform with an
Nvidia Quadro RTX A6000 GPU and Intel Core i9 13900K 24-core processor and the
Torch environment. PIP-Net was trained using the RMSProp optimiser. 256 hidden
units were used for the GRUs, and the sigmoid ($\sigma$) activation function
was applied to the GRUs for handling spatial kinematic data. To mitigate
overfitting, a dropout rate of 0.5 was introduced after the attention block.
Additionally, an L2 regularisation term of 0.0001 was incorporated into the
last fully connected layer. A stride of 3 steps is used for each input
sequence in the observation scene, resulting in a total of 10 frames for one
second. This stride reduces frame redundancy and compensates for feature
extraction time delay.

PIP-Net-$\alpha$: This model is designed for a single camera setup. The model
is trained on the PIE dataset and also evaluated against the PIE test set and
Frontal-Urban-PIP datasets. The model doesn’t have the Camera Index pipeline
within its architecture. Thereby, the outputs of the Max-pooling 3D (MP3D in
Figure 3) are directly forwarded to the flattened block, and the Shift blocks
are deactivated. A learning rate of $5\times 10^{-5}$ was used for 300 epochs
with a batch size 10. The model was tested on the Frontal-Urban-PIP dataset to
evaluate its generalisability. The input length stride was set to 1 because
the dataset’s FPS was already equal to 10.

PIP-Net-$\beta$: This model is designed for multi-camera setups to be
evaluated against Urban-PIP with three cameras. The training of this model is
performed using the Urban-PIP dataset with various observation times, a
learning rate of $4\times 10^{-5}$ across 400 epochs, and a batch size of 6.
The split ratio for training and testing samples is 80% (1,181 samples) and
20% (296 samples) of the dataset, respectively.

TABLE II: SOTA Performance comparison on the PIE dataset.
---
Model | Acc | AUC | F1 | Precision | Recall
ATGC (2017) | 0.59 | 0.55 | 0.39 | 0.33 | 0.47
Multi-RNN (2018) | 0.83 | 0.80 | 0.71 | 0.69 | 0.73
SingleRNN (2020) | 0.81 | 0.75 | 0.64 | 0.67 | 0.61
SFRNN (2020) | 0.82 | 0.79 | 0.69 | 0.67 | 0.70
PCPA (2021) | 0.87 | 0.86 | 0.77 | 0.75 | 0.79
CAPformer (2021) | 0.88 | 0.80 | 0.71 | 0.69 | 0.74
PPCI (2022) | 0.89 | 0.86 | 0.80 | 0.79 | 0.81
GraphPlus (2022) | 0.89 | 0.90 | 0.81 | 0.83 | 0.79
MCIP (2022) | 0.89 | 0.87 | 0.81 | 0.81 | 0.81
CIPF (2023) | 0.91 | 0.89 | 0.84 | 0.85 | 0.83
PIP-Net-$\alpha$ (Ours) | 0.91 | 0.90 | 0.84 | 0.85 | 0.84
Figure 6: Comparative analysis of prediction performance is conducted for
future time points in the study. Models are evaluated in terms of their
predictive accuracy from 1 second to 4 seconds ahead.

###  IV-C Comparative Results

Table II highlights the performance of our method on the PIE dataset. The
observation time ($m$) has been set to 16 frames, the same as the previous
methods to ensure a fair comparison with other methods. PIP-Net-$\alpha$,
achieves the highest values in all metrics, showcasing its prowess in
capturing the crossing intention classification. These metrics include
accuracy (Acc), precision, and recall rate, which quantify the model’s ability
to accurately predict the binary classification task. Additionally, the area
under the ROC curve (AUC), indicates the model’s proficiency in distinguishing
between different classes, and F1 score, represents the harmonic mean of
precision and recall rate.

In comparison to MultiRNN [24], SingleRNN [25], and SFRNN [23] models, which
use a CNN encoder for visual features and RNN-based encoder-decoder structure,
PIP-Net-$\alpha$ shows significant improvements by considering a new
combination of input features. Transformer and graph-based architectures used
in CAPformer [30] and GraphPlus [35] models have been less effective compared
to PIP-Net with RNN-based architecture. The proposed feature fusion approach
using seven input features, including three kinematic features and four
contextual features, has also improved AUC by +1% compared to CIPF [11], which
integrates eight distinct input features derived from pedestrians and vehicles
through three fusion modules.

####  IV-C1 Crossing time prediction

Depending on the traffic scenario, the model’s prediction performance may
vary. The model can predict the pedestrians’ estimated time to cross (ETC), 1
to 4 seconds in advance. For example, an ETC = 2 means the model expects or
predicts the target pedestrian crosses in 2 seconds.

We evaluated the performance of the proposed PIP-Net-$\alpha$ model across
various ETCs from 1 to 4 seconds. Figure 6 presents a comparison of the Acc,
AUC, and F1 performance of PIP-Net-$\alpha$ with two recent prediction models,
CIPF and MCIP, using the PIE dataset. All three models exhibited a decline in
performance across all metrics as the Estimated Time to Cross (ETC) increased,
i.e. when the models tried to have a longer-term prediction. Notably, the most
significant drop in AUC occurred between the ETC = 1-second and ETC = 2, with
MCIP and CIPF models decreasing by 6.8% and 6.7%, respectively. This is while
our model shows a 6.6% decrease in the AUC from ETC = 1 to 2. The Accuracy
also shows gradual decreases from the 3-second to the 4-second interval, with
all models experiencing a decrease of approximately 1% (MCIP) and 2% (CIPF),
respectively. As can be seen, the proposed model with green dashed lines
consistently outperformed the other models for all ETCs.

TABLE III: PIP-Net-$\alpha$ sub-variants using different input features on the
PIE dataset. Abbreviations are: LM: Local Motion, GM: Global Motion, GD:
Global Depth, and CD: Categorical Depth. The up arrow indicates an improvement
and the down arrow indicates a decrement in comparison with the baseline
variant.
---
Model | GM | LM | GD | CD | Acc | AUC | F1
$\alpha^{0}$ | - | - | - | - | 0.883 | 0.875 | 0.792
$\alpha^{1}$ | ✓ | - | - | - | 0.892$\uparrow$ | 0.881$\uparrow$ | 0.789$\downarrow$
$\alpha^{2}$ | - | ✓ | - | - | 0.889$\uparrow$ | 0.887$\uparrow$ | 0.801$\uparrow$
$\alpha^{3}$ | - | - | ✓ | - | 0.877$\downarrow$ | 0.870$\downarrow$ | 0.798$\uparrow$
$\alpha^{4}$ | - | - | - | ✓ | 0.904$\uparrow$ | 0.892$\uparrow$ | 0.829$\uparrow$
$\alpha^{5}$ | ✓ | - | ✓ | - | 0.875$\downarrow$ | 0.871$\downarrow$ | 0.789$\downarrow$
$\alpha$ | - | ✓ | - | ✓ | 0.911 | 0.903 | 0.844

####  IV-C2 Features Importance

Recent studies [30, 10, 11, 12] have emphasised the importance and reliability
of primary features including Bounding Box, Body Pose, Local Content, Vehicle
Speed and Semantic Context, in their elaborate experiments. The baseline
model, $\alpha^{0}$, comprises these primary features.

Initial experiments have demonstrated that excluding the input of the Bounding
Box leads to 8.6% decrease in accuracy compared with the baseline. While
omitting the Body Pose parameter only reduces accuracy by 3.5%. This
represents less importance of body pose compared to the bounding box which may
seem counter-intuitive. However, our further investigations confirm the
bounding box data is notably more important and useful as it includes the
pedestrian’s moving trajectory and tracking history over time, providing
valuable spatio-temporal information. While body poses spatio-temporal
information is not very important. The body pose only seems to matter when the
pedestrian is about to cross the road in the last few frames before crossing.
Otherwise, the body pose in previous moments, such as when the pedestrian is
on the sidewalk, is redundant.

Interestingly, removing the Vehicle Speed feature results in a 3.8% drop in
accuracy, making it the second most important input. This aligns with the
findings of the study by [30], which states that a model trained with the ego-
vehicle speed tends to focus on the ego-vehicle speed adjustment (e.g.
deceleration) to learn the pedestrian intention, rather than learning to
predict the intention from the pedestrian behaviour.

TABLE IV: The comparison table for the Frontal-Urban-PIP dataset.
---
Model | Acc | AUC | F1 | Precision | Recall
ATGC | 0.52 | 0.51 | 0.35 | 0.32 | 0.44
Multi-RNN | 0.64 | 0.63 | 0.49 | 0.51 | 0.48
SingleRNN | 0.65 | 0.64 | 0.54 | 0.57 | 0.53
SFRNN | 0.65 | 0.65 | 0.55 | 0.58 | 0.53
PCPA | 0.62 | 0.60 | 0.58 | 0.51 | 0.47
PPCI | 0.63 | 0.61 | 0.59 | 0.52 | 0.47
CAPformer | 0.64 | 0.60 | 0.55 | 0.58 | 0.54
GraphPlus | 0.64 | 0.61 | 0.57 | 0.59 | 0.56
PIP-Net-$\alpha$ (Ours) | 0.73 | 0.71 | 0.69 | 0.70 | 0.68

Excluding Semantic Context leads to a 3.4% accuracy decrease as it includes
details about road layout such as sidewalk positioning and drivable zones. The
impact of removing Local Content is minor, causing accuracy to decrease by
1.7%. It appears to lack comprehensive features on pedestrians’ intentions,
given the wide variety in appearance and accessories they may have.

Table III shows a comparison of the two input features we have used in the
proposed model, which correspond to the scene motion and depth information.
The $\alpha^{2}$ demonstrates that the local motion feature can exhibit
superior performance when compared to the global motion feature (2d), which
includes optical flow analysis of the entire scene. While optical flow is
typically sensitive to any movement between consecutive frames, it appears
that local motion can provide a coarse-grained feature and treat more
concisely to account pedestrians’ velocity and direction of movement,
regardless of other irrelevant objects in the surroundings.

Regarding depth information features, the categorical depth feature proposed
stands out as the most effective standalone feature, as evidenced by the
results of $\alpha^{4}$ in Table III, highlighting the importance of
pedestrian group densities, their distances from the ego-vehicle, and
interactions with other road users in the traffic scene. Conversely,
$\alpha^{3}$ performs the weakest among the sub-variants when compared with
the baseline, which utilises the global depth heatmap of the entire scene
(2e). This underperformance may be attributed to the unstable estimation of
depth for irrelevant surrounding objects. This is addressed in the categorical
depth by focusing only on pedestrians and vehicles, and then applying per-
instance normalisation (as shown in Figure 4).

Finally, the optimal outcome is attained by taking into account both the local
motion and categorical depth map, leading to enhancements in the baseline
regarding Acc, AUC, and F1 score by 1.8%, 3.1%, and 6.1%, respectively.

####  IV-C3 Generalisation

The PIP-Net-$\alpha$ evaluation is presented against the SOTA methodologies on
the Frontal-Urban-PIP dataset in Table IV. Notably, the models have never seen
the scenarios in their training phase. The outcomes of other methods were
generated using the pre-trained weights they provided. Overall, the majority
of models demonstrate improvements over ATGC across various metrics, with each
model exhibiting its unique strengths. However, we witnessed a decrement in
terms of performance for PCPA, PPCI, CAPformer, and GraphPlus. As far as the
research curiosity demand, we investigated the architecture of these models,
and it turned out they are suffering from low-quality global context and body
pose features. This is caused by the feature extractor algorithms, i.e.,
semantic segmentation and pose estimator, they have used which hinder the
classifier from judging based on precise features.

####  IV-C4 View Angle Expansion

We explored the enhancement of the field of view using three cameras to enable
the autonomous vehicle to perceive a larger portion of its surroundings. For
this purpose, we train PIP-Net-$\beta$ with three different observation times
($m$) of 20, 30, and 40 frames. Subsequently, we examined how the prediction
performance evolves as we expand the ETC prediction horizon from 1 to 4
seconds. As depicted in Figure 7, the accuracy of crossing intention
prediction decreases as the ETC prediction expands. However, the accuracy
often increases with the enlargement of observation time. Intriguingly, when
ETC = 4 and $m=40$ the accuracy was lower compared to $m=30$. This discrepancy
arises from the model predicting pedestrians to be crossing based on long-term
observations when, in reality, they did not cross. This highlights the
observation that a pedestrian’s previous actions do not always accurately
indicate their future intentions, as they can change their mind and act in an
instant [3].

Figure 7: Comparison of the crossing intention accuracy (Acc) for PIP-
Net-$\beta$ models (in three cameras mode) across various future time
intervals, highlighting the disparities in performance at each observation
time ($m$). The results are based on the Urban-PIP dataset.

###  IV-D Observational Results

We present the qualitative results of the PIP-Net-$\alpha$ network in Figure 8
for Frontal-Urban-PIP datasets and in Figure 9 for the PIE dataset. The
intention is represented by a confidence bar, where higher values (reddish
colour) indicate a high probability of the pedestrian crossing. Pedestrians
without the intention to cross are depicted with greenish bounding boxes and
lower values on the confidence bar.

(a) A non-signalised junction

(b) A signalised junction with a green light

(c) A signalised junction with a red light

Figure 8: A representation of the proposed model’s analysis on traffic video
scenes of Frontal-Urban-PIP dataset. Three distinct images showcase
intersections and traffic light conditions with confidence bars positioned in
front of pedestrians, aligning with their body pose direction. These
confidence bars visually indicate the model’s confidence about the crossing
intention of the detected pedestrians for the upcoming 2-4 seconds into the
future.

We displayed frames from the past $t-90$ frames to the $t$ frame, where $t$
represents the decisive moment. As the frames change, the prediction results
for the pedestrian’s crossing intention also evolve. Some pedestrians are
predicted to continue crossing, while others are forecasted to transition from
not crossing to crossing, or vice versa, depending on the pedestrian’s
direction or situation. For instance, the proposed model can predict the
intention of the pedestrian in case 7, as shown in Figure 9, when the ego
vehicle is going to turn left and the pedestrian tends to cross. The
contextual information alongside pedestrian features like motions and
direction of movements, seems to be crucial for accurately predicting their
behaviour in a given context.

Figure 9: Qualitative results of PIP-Net-$\alpha,(m=30)$ on the PIE dataset.
The pedestrian with a crossing intention has shown a higher value in the
confidence bar. We demonstrate the past frame $t-90\thicksim t$ and the ground
truth as a time in the future ($t+\epsilon$) with dashed bounding boxes, where
red: cross, green: not cross.

##  V Conclusion

This paper presented a framework called PIP-Net for predicting pedestrian
crossing intentions in real-world urban self-driving situations. Two variants,
$\alpha$ and $\beta$, were introduced to support different camera setups. By
utilising both kinematic data and spatial features of the driving scene, the
proposed model employed recurrent and temporal attention-based methodology to
predict pedestrians’ future crossing intentions accurately. Through
quantitative and qualitative experiments on the PIE dataset, the proposed
model achieved state-of-the-art performance with a 91% accuracy and an 84%
recall rate.

The Urban-PIP was introduced as a new dataset for the pedestrian intention
prediction task, including various AV driving scenarios and comprehensive
annotations on a multi-sensory setup, thereby enabling a better future
investigation of crossing behaviour studies. Our model demonstrated a
generalisation capability when applied to the Urban-PIP dataset by +9%, +10%,
and 12% improvement compared with other models in terms of accuracy, AUC, and
F1 score, respectively. This was underlined by the scene feature extractors
employed in training our model.

To enhance the visual encoding of road users and their relative distances to
the ego vehicle, we introduced a categorical depth feature map. This, combined
with the local motion flow feature, provided salient information about the
dynamics of the scene. Our results reveal that they cumulatively enhanced the
accuracy and F1 score of the baseline model by +2.8% and +5.2%, respectively.
Additionally, we investigated the impact of expanding the view angle using
three cameras and enlarging prior observation frames.

Our algorithm achieved 85.4% accuracy in predicting pedestrian crossing
intentions 2 seconds in advance and 79.3% accuracy for predictions between 2
and 4 seconds in advance. However, the algorithm is sensitive to the quality
and precision of the input features, specifically, scene context and body pose
information.

We anticipate that this algorithm can effectively prevent traffic accidents
and protect vulnerable road users by foreseeing the crossing behaviour of
nearby pedestrians.

## References

  * [1] G. Yannis, D. Nikolaou, A. Laiou, Y. A. Stürmer, I. Buttler, and D. Jankowska-Karpa, “Vulnerable road users: Cross-cultural perspectives on performance and attitudes,” _IATSS research_ , vol. 44, no. 3, pp. 220–229, 2020.
  * [2] N. Sharma, C. Dhiman, and S. Indu, “Pedestrian intention prediction for autonomous vehicles: A comprehensive survey,” _Neurocomputing_ , 2022.
  * [3] A. Najmi, T. Waller, M. Memarpour, D. Nair, and T. H. Rashidi, “A human behaviour model and its implications in the transport context,” _Transportation research interdisciplinary perspectives_ , vol. 18, p. 100800, 2023.
  * [4] Z. Zhou, Y. Liu, B. Liu, M. Ouyang, and R. Tang, “Pedestrian crossing intention prediction model considering social interaction between multi-pedestrians and multi-vehicles,” _Transportation Research Record_ , p. 03611981231187643, 2023.
  * [5] A. H. Kalantari, Y. Yang, J. G. de Pedro, Y. M. Lee, A. Horrobin, A. Solernou, C. Holmes, N. Merat, and G. Markkula, “Who goes first? a distributed simulator study of vehicle–pedestrian interaction,” _Accident Analysis & Prevention_, vol. 186, p. 107050, 2023.
  * [6] B. Yang, W. Zhan, P. Wang, C. Chan, Y. Cai, and N. Wang, “Crossing or not? context-based recognition of pedestrian crossing intention in the urban environment,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 6, pp. 5338–5349, 2021.
  * [7] K. Muhammad, T. Hussain, H. Ullah, J. Del Ser, M. Rezaei, N. Kumar, M. Hijji, P. Bellavista, and V. H. C. de Albuquerque, “Vision-based semantic segmentation in scene understanding for autonomous driving: Recent achievements, challenges, and outlooks,” _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * [8] L. Chen, S. Lin, X. Lu, D. Cao, H. Wu, C. Guo, C. Liu, and F.-Y. Wang, “Deep neural network based vehicle and pedestrian detection for autonomous driving: A survey,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 6, pp. 3234–3246, 2021.
  * [9] M. Gulzar, Y. Muhammad, and N. Muhammad, “A survey on motion prediction of pedestrians and vehicles for autonomous driving,” _IEEE Access_ , vol. 9, pp. 137 957–137 969, 2021.
  * [10] J.-S. Ham, K. Bae, and J. Moon, “MCIP: Multi-stream network for pedestrian crossing intention prediction,” in _Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I_. Springer, 2022, pp. 663–679.
  * [11] J.-S. Ham, D. H. Kim, N. Jung, and J. Moon, “CIPF: Crossing intention prediction network based on feature fusion modules for improving pedestrian safety,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 3665–3674.
  * [12] R. Ni, B. Yang, Z. Wei, H. Hu, and C. Yang, “Pedestrians crossing intention anticipation based on dual-channel action recognition and hierarchical environmental context,” _IET Intelligent Transport Systems_ , vol. 17, no. 2, pp. 255–269, 2023.
  * [13] T. Zhang, X. Chen, Y. Wang, Y. Wang, and H. Zhao, “Mutr3d: A multi-camera tracking framework via 3d-to-2d queries,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4537–4546.
  * [14] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 206–213.
  * [15] A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, “PIE: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6262–6271.
  * [16] B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles, “Spatiotemporal relationship reasoning for pedestrian intent prediction,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3485–3492, 2020.
  * [17] I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Benchmark for evaluating pedestrian action prediction,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021, pp. 1258–1268.
  * [18] F. Schneemann and P. Heinemann, “Context-based detection of pedestrian crossing intention for autonomous driving in urban environments,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 2243–2248.
  * [19] J. Mei, A. Z. Zhu, X. Yan, H. Yan, S. Qiao, L.-C. Chen, and H. Kretzschmar, “Waymo open dataset: Panoramic video panoptic segmentation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 53–72.
  * [20] J. Gesnouin, S. Pechberti, B. Stanciulescu, and F. Moutarde, “Assessing cross-dataset generalization of pedestrian crossing predictors,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022, pp. 419–426.
  * [21] H. Razali, T. Mordan, and A. Alahi, “Pedestrian intention prediction: A convolutional bottom-up multi-task approach,” _Transportation research part C: emerging technologies_ , vol. 130, p. 103259, 2021.
  * [22] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1724–1734.
  * [23] A. Rasouli, I. Kotseruba, and J. K. Tsotsos, “Pedestrian action anticipation using contextual feature fusion in stacked rnns,” in _British Machine Vision Conference_ , 2020.
  * [24] A. Bhattacharyya, M. Fritz, and B. Schiele, “Long-term on-board prediction of people in traffic scenes under uncertainty,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4194–4202.
  * [25] I. Kotseruba, A. Rasouli, and J. K. Tsotsos, “Do they want to cross? understanding pedestrian intention for behavior prediction,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 1688–1693.
  * [26] J. Lorenzo, I. Parra, F. Wirth, C. Stiller, D. F. Llorca, and M. A. Sotelo, “RNN-based pedestrian crossing prediction using activity and pose-related features,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 1801–1806.
  * [27] A. Singh and U. Suddamalla, “Multi-input fusion for practical pedestrian intention prediction,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2304–2311.
  * [28] K. Saleh, M. Hossny, and S. Nahavandi, “Real-time intent prediction of pedestrians for autonomous ground vehicles via spatio-temporal densenet,” in _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 9704–9710.
  * [29] ——, “Spatio-temporal densenet for real-time intent prediction of pedestrians in urban traffic environments,” _Neurocomputing_ , vol. 386, pp. 317–324, 2020.
  * [30] J. Lorenzo, I. Parra, R. Izquierdo, A. L. Ballardini, Á. Hernández-Saz, D. F. Llorca, and M. Á. Sotelo, “CAPformer: Pedestrian crossing action prediction using transformer,” _Sensors (Basel, Switzerland)_ , vol. 21, 2021.
  * [31] S. Neogi, M. Hoy, K. Dang, H. Yu, and J. Dauwels, “Context model for pedestrian intention prediction using factored latent-dynamic conditional random fields,” _IEEE transactions on intelligent transportation systems_ , vol. 22, no. 11, pp. 6821–6832, 2020.
  * [32] D. Zhang, F. Shi, Y. Meng, Y. Xu, X. Xiao, and W. Li, “Pedestrian intention prediction via depth augmented scene restoration,” in _2021 5th CAA International Conference on Vehicular Control and Intelligence (CVCI)_. IEEE, 2021, pp. 1–6.
  * [33] Z. Fang and A. M. López, “Is the pedestrian going to cross? answering by 2d pose estimation,” in _2018 IEEE intelligent vehicles symposium (IV)_. IEEE, 2018, pp. 1271–1276.
  * [34] Z. Fang and A. M. López, “Intention recognition of pedestrians and cyclists by 2d pose estimation,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 11, pp. 4773–4783, 2020.
  * [35] P. R. G. Cadena, Y. Qian, C. Wang, and M. Yang, “Pedestrian graph+: A fast pedestrian crossing prediction model based on graph convolutional networks,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 11, pp. 21 050–21 061, 2022.
  * [36] D. Yang, H. Zhang, E. Yurtsever, K. A. Redmill, and Ü. Özgüner, “Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention,” _IEEE Transactions on Intelligent Vehicles_ , vol. 7, no. 2, pp. 221–230, 2022.
  * [37] M. Azarmi, M. Rezaei, T. Hussain, and C. Qian, “Local and global contextual features fusion for pedestrian intention prediction,” in _Artificial Intelligence and Smart Vehicles_ , 2023, pp. 1–13.
  * [38] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 076–10 085.
  * [39] M. Rezaei, M. Azarmi, and F. M. P. Mir, “3D-Net: Monocular 3D object recognition for traffic monitoring,” _Expert Systems with Applications_ , vol. 227, p. 120253, 2023.
  * [40] D. Maji, S. Nagori, M. Mathew, and D. Poddar, “YOLO-Pose: Enhancing YOLO for multi person pose estimation using object keypoint similarity loss,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 2637–2646.
  * [41] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2462–2470.
  * [42] Y. Zhou, H. Zhang, H. Lee, S. Sun, P. Li, Y. Zhu, B. Yoo, X. Qi, and J.-J. Han, “Slot-vps: Object-centric representation learning for video panoptic segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3093–3103.
  * [43] J. Watson, O. Mac Aodha, V. Prisacariu, G. Brostow, and M. Firman, “The temporal opportunist: Self-supervised multi-frame monocular depth,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1164–1174.

##  VI Biography

|  Mohsen Azarmi is a Ph.D. Student at the University of Leeds, Institute for
Transport Studies, UK. He holds a master’s degree in Artificial Intelligence &
Robotics and his main research direction and expertise are Computer Vision,
Deep Neural Networks, and multi-sensor data fusion with a particular focus on
pedestrian activity recognition, transportation and traffic safety, and 3D
scene modelling.
---|---
|  Mahdi Rezaei is an Associate Professor of Computer Science and ML and
Leader of the Computer Vision Research Group at the University of Leeds,
Institute for Transport Studies. He received his PhD in Computer Science from
the University of Auckland, with the Top Thesis Award in 2014. Offering 18
years of service and research experience in academia and industry, Dr Rezaei
has published 60+ journals and conference papers in top-tier venues. He is
also the Principal Investigator and lead Co-Investigator of multiple European,
UKRI, and EPSRC AV-related projects such as L3Pilot, Hi-Drive, Research
England, and MAVIS.
---|---
|  He Wang is an Associate Professor at the Department of Computer Science,
University College London (UCL) and a Visiting Professor at the University of
Leeds. He is the Director of High-Performance Graphics and Game Engineering
and Academic Lead of Centre for Immersive Technology. His current research
interest is mainly in computer graphics, vision and machine learning and
applications.
---|---
|  Sebastien Glaser is a Professor of Intelligent Transportation Systems at
the Center for Accident Research & Road Safety (CARS) at Queensland University
of Technology, Australia. He received his Ph.D. in Automatic and Control
(defining a driving assistance system in interaction with the driver) in 2004.
He has worked in the development of Automated Driving Systems in interaction
with other road users and
---|---