# Cyclist Intention Detection:
A Probabilistic Approach

Stefan Zernetsch, Hannes Reichert, Viktor Kress, Konrad Doll, and Bernhard
Sick S. Zernetsch, H. Reichert, V. Kress, and K. Doll are with the Faculty of
Engineering, University of Applied Sciences Aschaffenburg, Aschaffenburg,
Germany<EMAIL_ADDRESS><EMAIL_ADDRESS>viktor.kress@th-
ab.de<EMAIL_ADDRESS>Sick is with the Intelligent Embedded Systems
Lab, University of Kassel, Kassel, Germany<EMAIL_ADDRESS>
###### Abstract

This article presents a holistic approach for probabilistic cyclist intention
detection. A basic movement detection based on motion history images (MHI) and
a residual convolutional neural network (ResNet) are used to estimate
probabilities for the current cyclist motion state. These probabilities are
used as weights in a probabilistic ensemble trajectory forecast. The ensemble
consists of specialized models, which produce individual forecasts in the form
of Gaussian distributions under the assumption of a certain motion state of
the cyclist (e.g. cyclist is starting or turning left). By weighting the
specialized models, we create forecasts in the from of Gaussian mixtures that
define regions within which the cyclists will reside with a certain
probability. To evaluate our method, we rate the reliability, sharpness, and
positional accuracy of our forecasted distributions. We compare our method to
a single model approach which produces forecasts in the form of Gaussian
distributions and show that our method is able to produce more reliable and
sharper outputs while retaining comparable positional accuracy. Both methods
are evaluated using a dataset created at a public traffic intersection. Our
code and the dataset are made publicly available.

##  I Introduction

###  I-A Motivation

In future traffic scenarios, more and more automated vehicles will share the
road with vulnerable road users (VRU), such as pedestrians and cyclists. To
guarantee safe interaction, automated vehicles not only have to be able to
capture the current traffic environment, but they must also anticipate future
movements of all traffic participants. While other automated vehicles can
share their planned paths, the future trajectories of VRU have to be
estimated. However, since VRU can change their path quickly, trajectory
forecasts are error prone. Therefore, instead of deterministic positional
forecasts, we have to estimate regions within which the VRU will reside with a
certain probability. In this article, we present a method to forecast VRU
positions in the form of probability distributions using a weighted ensemble
forecast. By weighting outputs in the form of Gaussian distributions created
by motion type specific Bayesian neural networks, we are able to reduce
forecast errors and forecast more reliable and sharper distributions compared
to a single network. The used weights are created by a basic movement
detection method, based on motion history images (MHI) [1], which encompass
the cyclist movements over a short period of time within a single image. The
method distinguishes between certain motion states, such as wait or start.
Since we want to give statistical guarantees that the VRU resides within the
forecasted regions, we have to evaluate the reliability of the network
outputs. Building on a previous approach for forecasted normal distributions,
we present a universal method to evaluate the reliability of forecasted
probability distributions. In addition to reliability, sharpness, i.e.,
narrowness, of the forecasts is crucial for successful path planing since a
very wide, but reliable forecast could prohibit a planning algorithm from
choosing a safe path around the cyclist. Hence, we present a method to
evaluate the sharpness of forecasts.

###  I-B Related Work

The field of VRU intention detection has become an active field of research
over the past years. The developed methods can roughly be grouped into basic
movement detection, where VRU motion states such as wait or start ought to be
detected, and trajectory forecast, with the goal to anticipate future VRU
positions. Publications regarding basic movement detection often focus on a
specific scenario or movement type. In [2], four methods based on Gaussian
process dynamical models (GPDM) and features derived from dense optical flow
to detect pedestrians’ intention to stop at a curbside or continue walking
were proposed. GPDM in combination with human poses extracted from image
sequences to detect start intentions of pedestrians were introduced in [3] and
[4]. In [5], the authors used 3D poses of cyclists to detect starting motions
from a moving vehicle. In [6] and [7], an approach based on MHI in combination
with a support vector machine (SVM) to detect pedestrians’ starting and
stopping intentions was presented. In both cases, the MHI based approach
outperformed an approach based on a Kalman filter with interacting multiple
models. In [8], the MHI based approach was adapted for cyclist starting
movement detection and extended by a residual convolutional neural network
(ResNet), where the ResNet outperformed the SVM approach. In this article, the
approach from [8] is adapted to a wide angle stereo camera system and extended
to cover all possible movement types of cyclists.

VRU trajectory forecast can be grouped into deterministic and probabilistic
forecast methods. Deterministic approaches usually rely on past VRU positions
to forecast future positions without quantization of uncertainties. For
example, in [9], the authors used GPDM with human poses to forecast pedestrian
trajectories up to one second into the future. In [10], the authors proposed a
method to forecast pedestrian trajectories using polynomial approximation of
the pedestrians’ past velocity in combination with a multi layer perceptron
(PolyMLP). The approach is extended to cyclist trajectories and compared to
two physical models in [11], where the MLP outperformed both approaches. In
[12] and [13], basic movement detections are used to improve the accuracy of
trajectory forecasts by training specialized forecast models and weighting the
outputs with pseudo-probabilistic outputs from basic movement detections. By
using the PolyMLP approach from [10] in combination with the MHI based
approach from [7], the authors were able to slightly improve the positional
accuracy of pedestrian trajectory forecasts. In [14] a cooperative basic
movement detection approach based on smart device sensor data and measurements
from a camera network was used. By weighting two specialized forecasting
models based on [10], the authors were able to reduce the forecast error of
starting motions by 21%.

Although these methods show promising results, to serve as basis for motion
planning algorithms in automated vehicles, the uncertainty of positional
forecast needs to be estimated. This is often achieved through prediction of
regions instead of positions in the form of probability distributions. In
[15], the method from [10] was enhanced by an uncertainty estimate created by
an unconditional model. The uncertainty estimates were used with a planning
method based on predictive control. In [16], the authors performed set-based
predictions using reachability analysis that provide bounded regions that
include all possible future positions of pedestrians ensuring the safety of
planned motions. In [17], the authors proposed a method to forecast
trajectories of pedestrians in a crowded area, taking into account the
interaction between different pedestrians. A long short-term memory model was
used to forecast positions in the from of bivariate Gaussian distributions,
the model was trained by minimizing the negative log-likelihood. Recurrent
neural networks in combination with context information to forecast cyclist
position in the form of Gaussian distributions were used in [18]. While all
the methods mentioned above create probabilistic forecasts, the reliability of
the produced outputs is not verified. In [19], a similar approach to forecast
cyclist trajectories is used and a method to evaluate the reliability of the
forecasted distributions is proposed. However, the method is limited to
unimodal Gaussian distributions. The article also shows that modeling future
trajectories with unimodal Gaussian distributions leads to underconfident
estimations. In this work, we extend the approach from [19]. By performing
basic movement detection to generate weights for different movement types and
training movement specific forecast models, we are capable of forecasting
multimodal distributions. Furthermore we extend the evaluation approach from
[19] for arbitrary distributions and show that our approach outperforms [19]
in terms of reliability.

In addition to reliability, which assesses the correctness of forecasted
probability densities, we need to evaluate sharpness, i.e., narrowness, of the
forecasted distributions. To the best of our knowledge, there has been no
evaluation of sharpness of distributions in the field of intention detection
so far. The concept of sharp probability distributions was described in [20].
We adapted the method to our approach.

While there has been some research in the field of pedestrian intention
detection, cyclists are mostly neglected in the field. Therefore, we focus on
the detection of cyclist intention, however, our approaches can be transfered
to other VRU as well.

###  I-C Main Contributions and Outline of this Paper

Our main contribution is a holistic approach for probabilistic cyclist
intention detection. By combining a basic movement detection with a
probabilistic trajectory forecast, we are able to produce reliable and sharp
estimates about the possible future positions of cyclists. Our basic movement
detection method creates two-channel MHI using instance segmentations,
capturing the movement of cyclist and bicycle separately. In addition, we
model all possible movements of cyclists. The probabilistic trajectory
forecast is based on a mixture of Gaussians by creating specialized
forecasting models for different movement types and weighting their outputs
using the probabilities of the basic movement detection. To evaluate our
results, we developed two methods which are capable of rating the reliability
and sharpness of arbitrary distributions. By using the method from [19] as
baseline, we show that our algorithm is able to produce more reliable and
sharper forecasts with comparable positional accuracy. Both methods are
evaluated using a dataset, which we created at an urban intersection in real
world traffic scenarios. The dataset and our code is made publicly available
[21][22].

The remainder of this article is structured as follows: In Sec. II-A, the
dataset and the research intersection where the dataset was created are
described. Sec. II-B outlines the overall intention detection process. Basic
movement detection and probabilistic trajectory forecast are described in Sec.
II-C and Sec. II-D, respectively. In Sec. III, we evaluate the results of our
trained models, followed by a conclusion and short outlook in Sec. IV.

##  II Method

###  II-A Test Site and Dataset

Figure 1: Probabilistic intention detection pipeline: a) Images
$I_{1/2;t}(u,v)$ are captured using wide angle stereo camera sytem and passed
to b) and c); b) Person and bicycle are segmented in images to create
$MHI_{1/2;t}(u,v,c)$. MHI is passed to d); c) Cyclist head trajectories in ego
coordinates ${}^{e}T_{in;t}$ are extracted and passed to d) and e). Ground
truth trajectory ${}^{e}T_{gt;t}$ is passed to e) during training; d) Basic
movement detection is performed. Resulting probabilities $p_{s;t}$ are passed
as weights to f); e) Probabilistic trajectory forecast is performed
individually for every specialized forecast model. Results in the form of
density functions $\mathcal{F}_{s;t}$ are passed to f); f) Density functions
$\mathcal{F}_{s;t}$ are weighted using $p_{s;t}$ to generate ensemble
forecasts.

In this section we provide a description of the test site where the dataset is
created, as well as a description of the dataset itself.

The dataset was created at the research intersection of the University of
Applied Sciences Aschaffenburg [23] using a wide angle stereo camera system
(Fig. 1a). The intersection located at the university consists of four arms,
two arms with 5 vehicle lanes each, one arm with three lanes, and one with two
lanes. With up to 24,000 vehicles a day, the intersection is strongly
frequented. To gather video data of VRU, a stereo camera system was mounted to
capture a corner of the intersection with a side walk, two pedestrian
crossings, and a bike lane. The full HD gray scale cameras 1 and 2 are mounted
approx. $5\text{\,}\mathrm{m}$ above ground at an angle of approx. $None$
relative to each other. Using this setup, most occlusions by vehicles or other
VRU can be avoided. The cameras operate at 50 fps.

The dataset consist of scenes containing cyclists moving across the research
intersection. We recorded 672 scenes where different cyclists were instructed
to move between certain end points at the intersection, while following the
traffic rules, and 976 scenes of uninstructed cyclists, resulting in 1,648
scenes in total. As basis for our developed methods, we recorded images
$\text{I}_{1/2;t}(u,v)$ from cameras 1 and 2, with pixel positions $u$ and
$v$, at the current time step $t$. These images are used to extract MHI and
trajectories (described in Sec. II-B), which we made publicly available [22].

###  II-B Intention Detection

In this section, we describe the overall intention detection process. We
define intention detection as umbrella term which encompasses detection of
motion states, such as wait or start, which we define as basic movement
detection, and the forecast of the cyclist positions in the form of
probability distributions, which we define as probabilistic trajectory
forecast.

Fig. 1 describes the intention detection process. In the first step (Fig. 1a),
the images $\text{I}_{1/2;t}(u,v)$, with pixel positions $u$ and $v$, are
grabbed from the cameras 1 and 2 and passed to the MHI generation (Fig. 1b)
and trajectory extraction (Fig. 1c).

To generate MHI, the cyclist is detected and tracked in both camera images. An
instance segmentation is performed on both images to detect cyclists and
bicycles. The intensity values of the cyclist and bicycle silhouettes created
by instance segmentation from the past $M$ time steps are decayed over time
and stacked resulting in $\text{MHI}_{1/2;t}(u,v,c)$, with two image channels
$c$ containing separate motion histories of cyclists and bikes.

For trajectory extraction, the head of the cyclist is detected in images of
both cameras at every time step. The head detections are then triangulated in
order to receive the head position ${}^{w}\vec{y}_{t}=[^{w}x_{t},^{w}y_{t}]$
in world coordinates (denoted by w) ${}^{w}x_{t}$ and ${}^{w}y_{t}$ (Fig. 1c).
As input for both basic movement detection and trajectory forecast, we use the
past 50 head detections, resulting in a trajectory
${}^{w}T_{in;t}=\\{^{w}\vec{y}_{t},^{w}\vec{y}_{t-$0.02\text{\,}\mathrm{s}$},^{w}\vec{y}_{t-$0.04\text{\,}\mathrm{s}$},...^{w}\vec{y}_{t-$1\text{\,}\mathrm{s}$}\\}$
consisting of the cyclist’s head positions over an input time set
$H_{\text{input}}=\\{$0\text{\,}\mathrm{s}$,$-0.02\text{\,}\mathrm{s}$,$-0.04\text{\,}\mathrm{s}$,...,$-1\text{\,}\mathrm{s}$\\}$
over the past second. To train the trajectory forecast network, the future
trajectory
${}^{w}T_{gt;t}=\\{^{w}\vec{y}_{t+$0.1\text{\,}\mathrm{s}$},^{w}\vec{y}_{t+$0.2\text{\,}\mathrm{s}$},...^{w}\vec{y}_{t+$2.5\text{\,}\mathrm{s}$}\\}$
with a forecast time set
$H_{\text{forecast}}=\\{$0.1\text{\,}\mathrm{s}$,$0.2\text{\,}\mathrm{s}$,...,$2.5\text{\,}\mathrm{s}$\\}$,
is used as ground truth. Before the trajectories are passed to the forecast
model, a transformation of the positions from world to ego coordinates is
performed:

${}^{e}\vec{y}_{t+h}=\begin{bmatrix}\cos(\varphi_{t})&\sin(\varphi_{t})\\\
-\sin(\varphi_{t})&\cos(\varphi_{t})\end{bmatrix}\cdot(^{w}\vec{y}_{t+h}-^{w}\vec{y}_{t}),$
(1)

with $\varphi_{t}$ being the movement direction of the cyclist at the current
time $t$ and $h\in H_{\text{input}}\cup H_{\text{forecast}}$ the relative time
offset from t. The transformation is performed for both, ${}^{w}T_{in;t}$ and
${}^{w}T_{gt;t}$, resulting in ${}^{e}T_{in;t}$ and ${}^{e}T_{gt;t}$, which
are passed to the probabilistic trajectory forecast (Fig. 1e).
${}^{w}T_{in;t}$ is passed to basic movement detection (Fig. 1d).

Basic movement detection is performed using $\text{MHI}_{1;t}(u,v,c)$,
$\text{MHI}_{2;t}(u,v,c)$, and ${}^{e}T_{in;t}$. It generates a probability
estimate $p_{s;t}$ for every possible motion state $s$ at current time $t$
(Fig. 1d).

The probabilistic trajectory forecasts of specialized models (Fig. 1e) are
used to generate positional forecasts in the form of Gaussian distributions
$\mathcal{F}_{s;t}$. We use individual models, each specifically trained for
one motion state $s$ of the cyclist.

In the final step (Fig. 1f), the specialized forecasts $\mathcal{F}_{s;t}$ are
weighted using basic movement detection probabilities $p_{s;t}$ and are
combined to one single distribution.

###  II-C Basic Movement Detection

In this section, the detection of basic movements using MHI and past
trajectories of cyclists is explained. In Sec. II-C1, we define the set of
basic movements we aim to detect, Sec. II-C2 describes how the MHI are
generated, in Sec. II-C3, we present our detection method including the
network architecture, and Sec. II-C4 describes the methods and metrics we used
to evaluate the detection algorithms.

####  II-C1 Basic Movements

The overall goal of basic movement detection is to detect motion states of
cyclists during which they show similar behavior. Using these detected states,
we aim to achieve more precise forecasts of the cyclists’ future trajectories
compared to forecasts without additional information about their current
motion state. We grouped every sample of our dataset $\mathit{D}$ into sets of
basic movements: move: cyclist moves with nearly constant velocity, stop:
cyclist slows down to a halt, wait: cyclist stands still until start, start:
cyclist begins to move after wait, and left/right: cyclist makes a left/right
turn. The basic movements start, stop, and move can be active simultaneously
with left/right. However, to train a unique trajectory forecast model for
every basic movement, we resolve this by grouping the movements into
wait/motion, indicating whether the cyclist is waiting or moving and
turn/straight, indicating whether the cyclist is turning or going straight. We
added three superstates motion, turn, and straight. The complete dataset
$\mathit{D}$ is divided into corresponding subsets as follows:
$\mathit{motion}=\\{x|x\in\mathit{D}\land x\notin\mathit{wait}\\}$,
$\mathit{turn}=\\{x|x\in\mathit{motion}\land(x\in\mathit{left}\lor
x\in\mathit{right})\\}$, and $\mathit{straight}=\\{x|x\in\mathit{motion}\land
x\notin\mathit{turn}\\}$. The corresponding state machine is visualized in
Fig. 2. Using this state machine, we avoid basic movements with multiple
active states at the same time, by adding samples where the cyclist is turning
exclusively to the turn set, regardless of the current movement state in
straight. Instead of detecting one basic movement for every time step and
choosing an according forecast model, we aim to assign a probability estimate
$p_{s;t}$ for every basic movement in
$S=\\{\textit{wait},\textit{start},\textit{stop},\textit{move},\textit{left},\textit{right}\\}$
and use the resulting probabilities $p_{s;t}$ with $s\in S$ to weight the
specialized forecast models.

Figure 2: Cyclist motion state machine.

####  II-C2 Generation of Motion History Images

In this section, we describe the generation of the MHI used for basic movement
detection. To generate an MHI, the silhouette of a cyclist including bicycle
is detected over the past $M$ time steps. The color intensity values of the
silhouettes are decayed over time, meaning silhouettes from earlier time steps
become darker then silhouettes from later time steps, and combined into one
image, encoding the past movements of the cyclists. Fig. 1b depicts the
consecutive steps of the MHI generation.

In the first step, a region of interest (ROI) enclosing the cyclist is
generated. The cyclist is detected in the image at the current time $t$ using
a pretrained Mask R-CNN with Inception ResNet [24]. The output of the network
consists of object boxes and instance segmentation masks for different
classes. To create a bounding box for the cyclist, the detected boxes for the
classes person and bike are used. The detected boxes are associated to a
cyclist using a simple nearest neighbor strategy. The person box is combined
with the bike box to create a bounding box enclosing the cyclist including the
bicycle. To account for movements of the cyclist from previous time steps, the
size of the resulting bounding box is enlarged by a factor $f_{b}$ to create
the ROI. The instance segmentation masks of the past time steps are saved as
binarized two channel images $\text{I}_{seg;t}(u,v,c)$ to generate the MHI.
The first channel of $\text{I}_{seg;t}(u,v,c)$ contains the instance
segmentation of the class person, the second contains bicycle, respectively.

In the second step, the cyclist instance segmentations of the past $M$ steps
are extracted from the segmentation images $\text{I}_{seg;t}(u,v,c)$ using the
ROI created in the first step and they are resized to the width and height $W$
and $H$ of the MHI, resulting in a sequence of cyclist and bike masks within
the ROI $I_{\text{MHI};t}(u,v,c)$. The color intensity values of the detected
masks are decayed using the decay value $\tau=\frac{M-i}{M}$, with $i$ being
the $i$-th time step, and are stacked to an MHI using Algorithm 1. To
distinguish between movements of the cyclists and movements of the bicycle, we
use a two channel MHI, with one channel containing the MHI of the person and
one channel containing the MHI of the bicycle silhouette. The resulting MHI is
depicted in Fig. 1b. This process is performed for both cameras resulting in
$\text{MHI}_{1;t}(u,v,c)$ and $\text{MHI}_{2;t}(u,v,c)$.

Algorithm 1 MHI Generation

1:$I_{\text{MHI};t}(u,v,c)\leftarrow$ cyclist/bike mask sequence at current
time t, with pixel positions $u$, $v$, and image channels $c$

2:$M\leftarrow$ number of time steps encoded in ${MHI}_{t}(u,v,c)$

3:$W\leftarrow$ image width

4:$H\leftarrow$ image height

5:$\Delta t:=\ $$0.02\text{\,}\mathrm{s}$ $\leftarrow$ duration between time
steps

6:$C=2\leftarrow$ number of channels

7:$MHI_{t}(u,v,c):=0$

8:for $i=M-1\text{ to }0$ do $\triangleright$ iterate over all images,

9:start with oldest

10: $\tau(i)=\frac{M-i}{M}$$\triangleright$ calculate decay value $\tau$

11: for $c=0\text{ to }C-1$ do$\triangleright$ repeat for both channels

12: for $u=0\text{ to }W-1$ do$\triangleright$ go through all pixels

13: for $v=0\text{ to }H-1$ do

14: if $I_{s;t-(i\cdot\Delta t)}(u,v,c)==1$ then

15: $MHI_{t}(u,v,c)=$ $\triangleright$ update MHI

16: $\tau(i)\cdot I_{s;t-(i\cdot\Delta t)}(u,v,c)$

####  II-C3 Network Architecture

In this section, the method to detect basic movements using MHI is described.
Rather than predicting one vector containing probabilities for each basic
movement, we use four different models to predict the probabilities for
wait/motion $\vec{p}_{wm}=\left[p_{wait},p_{motion}\right]$, straight/turn
$\vec{p}_{st}=\left[p_{straight},p_{turn}\right]$, left/right
$\vec{p}_{lr}=\left[p_{left},p_{right}\right]$, and start/stop/move
$\vec{p}_{ssm}=\left[p_{start},p_{stop},p_{move}\right]$, to generate a model
of the state machine in Fig. 2. By using separate models, we are able to train
specialized classifiers for each part of the state machine.

The specialized classifiers are trained using MHIs from both cameras in
combination with the past cyclist trajectory containing the last 50 positions
(i.e., $1\text{\,}\mathrm{s}$, see Sec. II-A) as network input and the
corresponding ground truth vector
$\vec{l}_{wm;t}=\left[l_{wait},l_{motion}\right]$,
$\vec{l}_{ts;t}=\left[l_{turn},l_{straight}\right]$,
$\vec{l}_{lr;t}=\left[l_{left},l_{right}\right]$, or
$\vec{l}_{ssm;t}=\left[l_{start},l_{stop},l_{move}\right]$ containing the
basic movement labels (zero or one) is used as network output.

Fig. 3 describes the architecture of the classification networks. Both MHI are
fed into CNN architectures (Fig. 3a and b). Since the ResNet architecture has
proven to perform well in image classification tasks [25], we decided to use
ResNet as CNN architecture. The past trajectory is fed into a fully connected
architecture (Fig. 3c). The outputs $\vec{h}_{1}$, $\vec{h}_{2}$, and
$\vec{h}_{3}$ are concatenated (Fig. 3d) and fed into another fully connected
architecture (Fig. 3e) followed by a fully connected layer with linear
activation with its output size being the number of probabilities in the
output vector $\vec{p}_{bm;t}$ and a softmax layer to generate the output
probabilities, so that $||\vec{p}_{bm;t}||_{1}=1$. With
$bm\in\\{wm,st,lr,ssm\\}$ describing the sub state machine with states
$S_{wm}=\\{\mathit{wait},\mathit{motion}\\}$,
$S_{st}=\\{\mathit{straight},\mathit{turn}\\}$,
$S_{lr}=\\{\mathit{left},\mathit{right}\\}$, and
$S_{ssm}=\\{\mathit{start},\mathit{stop},\mathit{move}\\}$. To be able to use
the predicted values as weights for specialized trajectory forecasts, we need
to estimate probabilities, which we achieve by applying a cross-entropy loss
function:

$L_{ce,bm}(\vec{p}_{bm;t},\vec{l}_{bm;t})=-\sum_{s\in
S_{bm}}l_{s;t}\cdot\log(p_{s;t}),$ (2)

with $\vec{l}_{bm;t}$ containing the ground truth values $l_{s;t}$ for every
state $s$ in $S_{bm}$ and $\vec{p}_{bm;t}$ the predicted probabilities
$p_{s;t}$, respectively. To improve the reliability of the predicted
probabilities, we perform calibration using isotonic regression [26]. Both
calibrated and uncalibrated probabilities are evaluated.

Figure 3: Basic movement detection network architecture.

####  II-C4 Evaluation Method

To evaluate basic movement detection results, we look at the classification
results of individual time steps. However, since the overall goal is to
generate weights for specialized forecast models, we also have to evaluate the
generated probabilities.

To evaluate the classification results of individual time steps, we calculate
the $\text{F}_{1}$-scores of the classifiers for every sub-state machine.
Since we have multi class classification problems, we compute both, the micro
averaged and macro averaged $\text{F}_{1}$-scores:

$\text{micro-
precision}=\frac{\sum_{c=1}^{N_{c}}tp_{c}}{\sum_{c=1}^{N_{c}}tp_{c}+\sum_{s=1}^{N_{c}}fp_{s}},$
(3) $\text{micro-
recall}=\frac{\sum_{c=1}^{N_{c}}tp_{c}}{\sum_{c=1}^{N_{c}}tp_{s}+\sum_{c=1}^{N_{c}}fn_{c}},$
(4) $\text{macro-
precision}=\frac{\sum_{c=1}^{N_{c}}\frac{tp_{c}}{tp_{c}+fp_{c}}}{N_{c}},$ (5)
$\text{macro-
recall}=\frac{\sum_{c=1}^{N_{c}}\frac{tp_{c}}{tp_{c}+fn_{c}}}{N_{c}},$ (6)

$\text{F}_{1,\text{micro}}=\frac{\text{micro-precision}\cdot\text{micro-
recall}}{\text{micro-precision}+\text{micro-recall}},$ (7)
$\text{F}_{1,\text{macro}}=\frac{\text{macro-precision}\cdot\text{macro-
recall}}{\text{miacro-precision}+\text{macro-recall}},$ (8)

where $N_{c}$ describes the number of classes, i.e., states in $bm$, $tp_{c}$
describes the number of true positives, $\textit{fn}_{c}$ and $fp_{c}$, the
numbers of false negatives and false positives of the $c$-th class,
respectively. Additionally, we visualize the results using confusion matrices
to illustrate misclassification between classes.

To evaluate the quality of the generated probabilities, in addition to the
log-likelihood (Eq. 2) we compute the Brier scores for every state $s\in S$:

$\text{BS}_{s}=\frac{1}{N_{t}}\sum_{i=1}^{N_{t}}(p_{s,i}-l_{s,i})^{2},$ (9)

with the predicted probability $p_{s,i}$ and label $l_{s,i}$ of state $s$ of
the i-th sample of all $N_{t}$ test samples. In contrast to the log-
likelihood, the Brier score is always between zero and one, with zero being
the best possible outcome, making it easier to interpret. To visualize the
reliability of the generated probabilities, we use Q-Q plots [27]. Predicted
probabilities are considered reliable, if a certain predicted probability $p$
leads to a correct classification in $p\cdot N$ times of all $N$ predictions,
e.g., for predicted probabilities of 0.5, the predictions have to be correct
in 50% of all cases. The Q-Q plots are generated for each sub-state machine.

###  II-D Probabilistic Trajectory Forecast

This section covers the probabilistic forecast of cyclist trajectories. In
Sec. II-D1, we describe the model from [19], on which the model in this work
is based and which is also used as baseline model. In Sec. II-D2, we extend
this model to a framework for motion state specific forecasts and use the
basic movement detection from Sec. II-C to properly model transitions between
movement states. Sec. II-D3 describes various metrics for the evaluation of
probabilistic forecasts.

####  II-D1 Probabilistic Forecast

In [19], a method which incorporates uncertainty estimates to deterministic
trajectory forecasts by modeling uncertainties with normal distributions is
proposed. Instead of forecasting only the most likely positions of a VRU, this
method can be used to forecast how likely it is that a VRU will be within a
certain region. These forecasts include the most likely positions at the means
of normal distributions and uncertainty estimates encoded in confidence
regions which in turn are described by covariance matrices. For reliable
forecasts these confidence regions give us the probability that the VRU will
reside within the region. Since we use one parametric normal distribution per
time horizon, we write the distribution as:

$\mathcal{N}_{t+h}(\vec{y})=\frac{exp(-\frac{1}{2}\overbrace{[\vec{y}-\vec{\mu}_{t+h}]^{T}\cdot\mathbb{S}^{-1}_{t+h}\cdot[\vec{y}-\vec{\mu}_{t+h}]}^{d_{t+h}^{2}})}{\sqrt{(2\pi)^{2}\det\mathbb{S}_{t+h}}},$
(10)

with $\vec{\mu}_{t+h}=\begin{bmatrix}\mu_{x;t+h}&\mu_{y;t+h}\end{bmatrix}$
being the mean, $\mathbb{S}_{t+h}$ the covariance matrix for a forecasted time
horizon $h\in H_{\text{forecast}}$ in longitudinal ($x$) and lateral ($y$)
direction, and $\vec{y}$ being an arbitrary point in space $\mathbb{R}^{2}$.
They are used to calculate the squared Mahalanobis distance $d_{t+h}^{2}$
between the mean of the distribution to an arbitrary point $\vec{y}$. The
covariance matrices and the means are parameterized by a neural network.
Therefore, they are functions of the input trajectory ${}^{e}T_{in;t}$ and the
network parameters $W$. The covariance matrix can be constructed from the
longitudinal and lateral standard deviations $\sigma_{x;t+h}$ and
$\sigma_{y;t+h}$, as well as a correlation coefficient $\rho_{t+h}$:

$\mathbb{S}_{t+h}=\begin{bmatrix}\sigma^{2}_{x;t+h}&\rho_{t+h}\sigma_{x;t+h}\sigma_{y;t+h}\\\
\rho_{t+h}\sigma_{x;t+h}\sigma_{y;t+h}&\sigma^{2}_{y;t+h}\end{bmatrix}.$ (11)

Figure 4: Specialized trajectory forecast network architecture.

The covariance matrix is a symmetric and positive definite matrix. In the
model implementation, the standard deviations $\sigma_{x;t+h}$ and
$\sigma_{y;t+h}$ must be greater than zero and the correlation coefficient
$\rho_{t+h}$ must be within $\left]-1,1\right[$. Therefore, we have to
transform the linear network outputs $\vec{o}_{\sigma}$ and $\vec{o}_{\rho}$
(Fig. 4) which are used to form the covariance matrices $\mathbb{S}_{t+h}$ to
match these constraints. For the correlation coefficient, we apply the
hyperbolic tangens $tanh$ element-wise to transform the corresponding outputs
$\vec{o}_{\sigma}$ into the desired interval. We also scale this interval to
$\left]-\epsilon_{\rho},\epsilon_{\rho}\right[$ since a high correlation can
lead to numerical problems. In a preliminary investigation we found that the
min/max values of the correlation coefficient, forecasted by the network are
$\left[-0.75,0.75\right]$, therefore we see $\epsilon_{\rho}=0.9$ as a valid
choice to model the distributions and suppress numerical instabilities. For
the standard deviations we apply the Softplus-Function
$\textit{softplus}(\vec{o}_{\sigma})=\ln(1+e^{\vec{o}_{\sigma}})$ element-wise
on the network outputs $\vec{o}_{\sigma}$ to obtain positive values. To
prevent any $\sigma$ from becoming zero, we add a small $\epsilon_{\sigma}$ to
the result of the Softplus-function. To learn the uncertainty estimates from
the given data, we minimize the average negative log-likelihood over the
forecasted time horizons, for every input and output trajectory pair, which
can be seen as a cross entropy minimization. We decompose the negative log-
likelihood of one forecasted time horizon into:

$\displaystyle\textit{NLL}_{t+h}$
$\displaystyle=\underbrace{\frac{1}{2}d_{t+h}^{2}(\vec{y}_{t+h},W,^{e}T_{in,t})}_{\textit{error}}$
(12)
$\displaystyle+\underbrace{\frac{1}{2}\log\det\mathbb{S}_{t+h}(W,^{e}T_{in,t})}_{\textit{residual}}$

In this decomposition, the error term describes the distance between the
forecasted distribution and a ground truth point $\vec{y}_{t+h}$ while the
residual term contains a volumetric measure of uncertainty encoded in the
covariance matrix. In an optimization scenario, the solver minimizes the error
term first, since this can also be achieved with higher uncertainty estimates.
The residual term keeps the uncertainty estimates as small as possible. Since
the squared Mahalanobis distance $d_{t+h}^{2}$ within the error term is based
on the matrix inverse and the residual term on the determinant of the
covariance matrix $\mathbb{S}_{t+h}$, this loss is prone to numerical issues,
especially during the gradient calculation. To resolve these issues, we invert
the matrix and build the determinant using the Cholesky decomposition. As
described in Sec. II-B, we use trajectories in ego coordinates. We optimize
the network on NLL averaged over all forcast horizons and all samples in the
training set. The forecasted distributions are given in ego coordinates. To
obtain distributions in world coordinates, the geometric transformation
described in Eq. 1 must be reversed.

####  II-D2 Motion Specific Forecast

In [19], it was discovered that a single normal distribution per forecast
horizon is not sufficient to reliably model the forecast uncertainty. It was
concluded that the model tends to output underconfident forecasts. To reduce
this effect, we provide a framework for motion specific forecasts in which
every motion state is modeled by forecasted distributions of a specialized
model:

$\mathcal{D}_{t+h}(\vec{y}_{t+h})=\sum_{s\in
S}p_{s:t}\cdot\mathcal{F}_{s;t+h}(\vec{y}_{t+h}).$ (13)

The joint distribution $\mathcal{D}_{t+h}$ consists of the forecasted
likelihood $\mathcal{F}_{s;t+h}$ of the individual basic movement $s$,
weighted by a probability $p_{s;t}$ for the respective motion state. For the
basic movements start, stop, left, right, and move the forecasted
distributions are normal distributions
$\mathcal{F}_{s;t+h}=\mathcal{N}_{s;t+h}$, parameterized by a neural network
as described in Sec. II-D1, specialized for the respective motion state.
Originally we handeled wait in the same way, but we found that the model
cannot describe the variances for this motion state sufficiently. Since
$\textit{softplus}(x)$, which we apply to transform the outputs used for the
standard deviation, decreases to zero exponentially for an $x$ smaller than
one, we assume this to be a combination of a vanishing gradient problem and
the fact that the future positions of wait cannot be sufficiently modeled by a
single Gaussian distribution. For wait we fit a Gaussian mixture model [28]
for every time horizon $h$ of the forecasts
$\mathcal{F}_{\textit{wait}:t+h}=\mathcal{GMM}_{\textit{wait}:t+h}$. We assume
this to be valid since the input is always very similar for samples of the
motion state wait and verified the validity with the methods of Sec. II-D3.
The forecasts for the motion state wait are static and do not depend on the
input trajectory ${}^{e}T_{in,t}$.

####  II-D3 Evaluation Methods

As discussed in [20], we should focus on two characteristics of the forcasting
model. One is the reliability and the second is the sharpness. While
reliability describes whether the conditional variance of a forecast is
correctly assessed in each situation, the sharpness measures the general
narrowness of the forecasted distributions. Reliability is an essential
property and sharpness can be used to decide how usable a model is in a real
world application. For our application, we want to forecast distributions
which are as sharp as possible, while retaining reliability, as they both
serve as a basis for planning algorithms of autonomous vehicles. In [19], the
authors evaluated the reliability of a forecasted distribution by plotting the
confidence interval derived from the forecasts against the observed frequency
of ground truth values within the confidence interval. This method is not
usable for arbitrary distributions, since the method used to calculate the
confidence interval is only applicable for normal distributions. We propose
the calculation of confidence sets and confidence levels instead of intervals
as a confidence measure for arbitrary distributions. For a given distribution,
we define a confidence set $\Omega(1-\alpha)$ as a set of points with the
confidence level of $1-\alpha$, with $\alpha\in\interval[open]{0.0}{1.0}$. To
obtain the confidence level $1-\alpha(\vec{y})$ of a point $\vec{y}$, we
search for points $\vec{z}\in\mathbb{R}^{2}$ which have a probability density
level greater than or equal to the density $\mathcal{D}(\vec{y})$.

Figure 5: Confidence level $1-\alpha(\vec{y})$ (solid curve) of $\vec{y}$
(dot), with $N$ random samples $\vec{z}\in\mathbf{Z}$ with
$\mathcal{D}(\vec{z})\geq\mathcal{D}(\vec{y})$ as + and
$\mathcal{D}(\vec{z})<\mathcal{D}(\vec{y})$ as x.

Doing this for every point in $\mathbb{R}^{2}$ is not feasible in practice, so
we draw $N$ random samples $\mathbf{Z}\sim\mathcal{D}$ from the distribution
$\mathcal{D}$ to find these points. The confidence level estimate
$1-\alpha(\vec{y})$ of the observation $\vec{y}$ is determined by:

$1-\alpha(\vec{y})=\frac{1}{N}\sum_{\vec{z}\in\mathbf{Z}}\begin{cases}1,&\text{
if }\mathcal{D}(\vec{z})\geq\mathcal{D}(\vec{y})\\\
0,&\text{otherwise}\end{cases}.$ (14)

Fig. 5 illustrates the confidence level estimation for an exemplary
distribution. By knowing the confidence level of arbitrary points, we can
build confidence sets $\Omega(1-\alpha)$ as a set of points with at least the
confidence level $1-\alpha$:

$\Omega(1-\alpha)=\big{\\{}\vec{y}|\vec{y}\in\mathbb{R}^{2}\land\overbrace{1-\alpha(\vec{y})}^{\text{est.
confidence level of }\vec{y}}\geq 1-\alpha\big{\\}}.$ (15)

We can evaluate the reliability of the forecasting framework by calculating
the estimated confidence level $1-\alpha(\vec{y})$ for every corresponding
pair of forecasted distribution and ground truth point. The observed relative
frequency of occurrences $f_{o}(1-\alpha)$ should match $1-\alpha$ itself. We
evaluate the reliability visually by plotting the observed frequency over
$1-\alpha$ which should ideally result in a line with a slope of one. As
comparative metrics we build $f_{o;t+h}(1-\alpha)$ from every corresponding
pair of forecasted distribution $\mathcal{D}_{t+h}$ and ground truth position
$\vec{y}_{t+h}$ in the validation set and search for the largest (Eq. 16) and
average (Eq. 17) distance from the ideal value $1-\alpha$ over all $\alpha\in
A=\\{0.01,0.02,...0.99\\}$ and all time horizons $h\in H_{\text{forecast}}$:

$\hat{\Gamma}=\max_{h}\max_{1-\alpha}|1-\alpha-f_{o;t+h}(1-\alpha)|$ (16)
$\bar{\Gamma}=\frac{1}{|H_{\text{forecast}}|\cdot|A|}\sum_{h\in
H_{\text{forecast}}}\sum_{\alpha\in A}|1-\alpha-f_{o;t+h}(1-\alpha)|$ (17)

For these metrics the perfect score is 0, which is a completely reliable
forecasting system, and the worst score is 1, which corresponds to an
unreliable system.

For a certain confidence level $1-\alpha$ of a distribution we define the
sharpness $\kappa(1-\alpha)$ as the volumetric measure of the confidence set
$\Omega(1-\alpha)$. Since building the confidence set over $\mathbb{R}^{2}$ is
analytically not feasible, a volumetric measure of the confidence sets and the
confidence sets themselves are determined through quasi-Monte Carlo
integration [29].

To provide a comparative metric for the sharpness of the forecasting system we
build $\bar{K}(1-\alpha)$ as the average sharpness of a specific time horizon
$\bar{\kappa}_{t+h}(1-\alpha)$, normalized to the forecast horizon:

$\bar{K}(1-\alpha)=\frac{1}{|H_{\text{forecast}}|}\sum_{h\in
H_{\text{forecast}}}\frac{\bar{\kappa}_{t+h}(1-\alpha)}{h}.$ (18)

To rate the positional accuracy, we use the Euclidean error between he most
likely point of the forecasted distribution, i.e., the mode
$\vec{\omega}_{t+h}$, and the ground truth $\vec{y}_{t+h}$ over every
corresponding pair of forecasted mode and ground truth position and calculate
the average Euclidean error $AEE_{t+h}$ for a specific time horizon over all
pairs of forecasted modes and ground truth points. As a comparative metric, we
use the ASAEE, introduced by Goldhammer [13], since it produces a single value
of the forecast error normalized to the forecast horizon
$H_{\text{forecast}}$:

$ASAEE=\frac{1}{|H_{\text{forecast}}|}\sum_{h\in
H_{\text{forecast}}}\frac{AEE_{t+h}}{h}.$ (19)

##  III Experimental results

###  III-A Basic Movement Detection

In this section, we present the results of the basic movement detection
described in Sec. II-C. To train the models, the dataset described in Sec.
II-A was split into training, validation, and test sets using a 60-20-20
split. Since the samples were created from scenes and therefore, are not
independent from each other, the split was performed by ensuring no samples
from the same scene were assigned to different sets. Additionally, the samples
were distributed to the sets in a way that each state $s$ has the same
percentage in each set (as far as possible). To choose between calibrated and
uncalibrated classifiers (Sec. II-C3), we created the Q-Q plots for all
classifiers using the calibrated and uncalibrated probabilities, chose the
best classifiers using the validation set, and picked the classifiers with the
best validation results. For wait/motion, turn/straight, and, left/right, the
calibrated classifiers showed the best validation results, for
start/stop/move, the uncalibrated classifiers showed better results. These
classifiers were used to create the results in Sec. III-B.

We first evaluate the classifiers’ overall accuracy using the micro and macro
$\text{F}_{1}$-scores (Tab. I). Additionally, the confusion matrices of the
best classifiers are depicted in Fig. 6. To compare the calibrated and
uncalibrated classifiers, we created outputs for all classifiers using the
test set. The wait/motion and left/right classifiers both perform very well.
In both cases misclassifications mainly occur at the transitions between the
two states. The confusion matrix of turn/straight shows that turn samples are
classified as straight in 20% of all cases. This often happens when the
cyclist moves on a curve with a large radius, as well as at the beginning of a
turn movement where the past trajectory and the MHI show only few signs of a
turning movement. Misclassifications of straight samples mostly occur at the
end of turn movements, where past trajectory and MHI still show signs a of
turn. By looking at the confusion matrix of start/stop/move, we can see that
start and stop are often predicted as move. Especially the beginning of stop
is hard to classify, since stop is only labeled if the cyclist slows down to a
halt. Other breaking maneuvers are labeled as move. If we compare the
$\text{F}_{1}$-scores of the calibrated and uncalibrated classifier, we see
only small differences.

Since the predicted probabilities are used to weight between specialized
forecasting models, in addition to the overall accuracy of the predicted
classes, we also have to evaluate the probabilities themselves. Therefore, we
created the BS (Tab. I) and Q-Q plots (Fig. 7) for the test set to rate the
accuracy and reliability of the probabilistic predictions. Since wait/motion,
turn/straight, and left/right are two-class classifications with
$p_{1}=1-p_{2}$, we only display the Q-Q plots for one of the classes, since
the Q-Q plot of the second class results in the inverted plot of the first
class. In the cases of wait/motion and turn/straight the calibration leads to
slight improvements in Brier scores and reliability. Although the calibration
of left/right leads to an improvement on the validation set, the calibrated
classifier shows worse results on the test set. This can be accounted to the
fact that the left/right classifier almost exclusively produces probabilities
close to 1.0 or 0.0. Therefore, few probabilities can be used for the
calibration. In the case of start/stop/move, the calibration does not lead to
an improvement for neither the validation nor the test set. However, the Q-Q
plots of start, stop, and move show that the results from the network are
already well calibrated.

TABLE I: Classification results.

Uncalibrated results
---
| wait/motion | turn/straight | left/right | start/stop/move
$\text{F}_{1,\text{micro}}$ | 0.979 | 0.927 | 0.989 | 0.853
$\text{F}_{1,\text{macro}}$ | 0.979 | 0.883 | 0.989 | 0.820
| wait | motion | turn | straight | left |  right | start | stop | move
$\text{BS}_{s}$ | 0.016 | 0.016 | 0.054 | 0.054 | 0.008 | 0.008 | 0.057 | 0.058 | 0.097

Calibrated results
---
| wait/motion | turn/straight | left/right | start/stop/move
$\text{F}_{1,\text{micro}}$ | 0.981 | 0.933 | 0.986 | 0.853
$\text{F}_{1,\text{macro}}$ | 0.981 | 0.886 | 0.985 | 0.820
| wait | motion | turn | straight | left |  right | start | stop | move
$\text{BS}_{s}$ | 0.015 | 0.015 | 0.050 | 0.050 | 0.012 | 0.012 | 0.073 | 0.077 | 0.128

Figure 6: Confusion matrices of different sub-state machines. Figure 7: Q-Q
plots of basic movement detection, with uncalibrated results (solid),
calibrated results (dotted), and ideal results (dashed).

###  III-B Probabilistic Trajectory Forecast

Figure 8: Reliability diagrams for the baseline (top row) and the ensemble
(bottom row) for the forecasted time horizons $0.5\text{\,}\mathrm{s}$
(dashed), $1.5\text{\,}\mathrm{s}$ (dotted) and $2.5\text{\,}\mathrm{s}$
(dash-dotted). Figure 9: Example forecast from the baseline (left) and from
the ensemble (right) approaches, with input trajectory (dark solid line),
ground truth trajectory (gray solid line), 68% levels (solid contours), 95%
levels (dashed contours), estimated modes (plus), and ground truth (dots on
trajectory) at forecasted time horizons $0.5\text{\,}\mathrm{s}$,
$1.5\text{\,}\mathrm{s}$, and $2.5\text{\,}\mathrm{s}$.

We trained one specific neural network for each motion state on data of start,
stop, left, right, and move each, as well as one general baseline neural
network on data of a merged set from all motion states. For the motion state
wait, we fitted a Gaussian mixture model for every forecast time horizon. To
train the models, the same split as in Sec. III-A was used.

To select the best configuration for the models, a parameter sweep was
performed. The models were trained for 500,000 training steps, where every
1,000-th step a validation was performed. The final networks were selected by
choosing the configurations with the smallest validation score. We then
trained the best network up to the step with the best validation score on a
combined set of training and validation data.

TABLE II: Probabilistic trajectory forecast results.

Baseline
---
| complete | wait | left | right | start | stop | move
$\hat{\Gamma}$ | 0.44 | 0.66 | 0.2 | 0.24 | 0.23 | 0.29 | 0.25
$\overline{\Gamma}$ | 0.2 | 0.29 | 0.06 | 0.05 | 0.02 | 0.1 | 0.07
$\overline{K}(0.68)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $0.62$ | $0.19$ | $1.9$ | $1.86$ | $1.37$ | $0.97$ | $1.52$
$\overline{K}(0.95)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $1.62$ | $0.5$ | $4.99$ | $4.87$ | $3.58$ | $2.55$ | $3.99$
$\overline{K}(0.99)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $2.48$ | $0.76$ | $7.61$ | $7.43$ | $5.47$ | $3.89$ | $6.09$
$ASAEE\text{ in }$\mathrm{m}$/$\mathrm{s}$$ | 0.21 | 0.09 | 0.55 | 0.57 | 0.46 | 0.32 | 0.42

Ensemble
---
| complete | wait | left | right | start | stop | move
$\hat{\Gamma}$ | 0.14 | 0.18 | 0.1 | 0.12 | 0.07 | 0.13 | 0.19
$\overline{\Gamma}$ | 0.03 | 0.05 | 0.05 | 0.03 | 0.03 | 0.02 | 0.02
$\overline{K}(0.68)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $0.33$ | $0.02$ | $1.06$ | $0.97$ | $0.76$ | $0.62$ | $1.2$
$\overline{K}(0.95)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $1.55$ | $0.92$ | $3.03$ | $2.98$ | $2.4$ | $2.03$ | $3.39$
$\overline{K}(0.99)\text{ in }$\mathrm{m}$^{2}/$\mathrm{s}$$ | $2.64$ | $1.65$ | $4.48$ | $5.03$ | $4.03$ | $3.42$ | $5.38$
$ASAEE\text{ in }$\mathrm{m}$/$\mathrm{s}$$ | 0.21 | 0.08 | 0.49 | 0.41 | 0.61 | 0.3 | 0.42

Tab. II shows the overall results of the probabilistic trajectory forecast.
Fig. 8 shows the reliability diagrams for the complete test set and for all
movement types separately for the baseline (top row) and ensemble (bottom row)
models. Compared to the baseline model, the ensemble achieved better overall
scores in both largest $\hat{\Gamma}$ and average $\overline{\Gamma}$ distance
to the ideal line (Fig. 8). $\hat{\Gamma}$ was reduced by 68%,
$\overline{\Gamma}$ by 85%. While we see an improvement for all motion states,
especially wait, stop, and move show large improvements. The baseline produced
underconfident forecasts, i.e., confidence regions are too wide compared to
the ground truth, for these motion states. We attribute this to the use of a
single Gaussian distribution per time step of the baseline, which is not
sufficient to describe the real distribution. The sharpness evaluation for the
complete dataset shows smaller values for $\overline{K}(0.68)$ and
$\overline{K}(0.95)$, and larger values for $\overline{K}(0.99)$ (see Tab.
II). However, except from the sharpness of wait, the sharpnesses of all other
motion states could be reduced by the ensemble, which is achieved through its
multi-modality. The larger $\overline{K}(0.99)$ of wait can be accounted to a
small quantity of wait samples $2.5\text{\,}\mathrm{s}$ before the beginning
of start, since these samples contain the movement of the start motion in
their ground truth trajectory. This behavior cannot be modeled by a single
Gaussian distribution. The ASAEE of baseline and ensemble show similar
results, while start of the ensemble shows worse results compared to the
baseline. Tests with ideal weights show an improvement of the ASAEE of the
ensemble over the baseline for start movements. Therefore, the larger ASAEE
can be attributed to late start detection. Fig. 9 shows an example forecast of
the baseline model (left) and the ensemble (right) for a left motion, showing
a sharper forecast with better mode estimation for the ensemble.

##  IV Conclusions and Future Work

In this article, we presented a new approach to perform probabilistic cyclist
intention detection. We compared our method to a baseline method and showed
that our approach is able to achieve more reliable and sharper forecasts while
retaining similar positional accuracies. When it comes to reliability we were
able to reduce the average reliability error by 85%. Sharpness was improved
for all motion types except wait.

In our future work, we will focus on improving basic movement detection, as it
is crucial for the overall intention detection performance. In this work we
assign a basic movement probability for every time step independently from
previous probabilities. However, by incorporating knowledge about possible
movement transitions in combination with detected states from previous steps,
the accuracy and robustness of basic movement detection could be improved. We
also plan to investigate ensemble methods to further improve basic movement
detection.

To further improve the sharpness of our forecasts, we will investigate the use
of different distribution functions for the specialized models, e.g., skew
normal distributions or Gaussian mixtures. We also plan to compare our method
to different approaches used to estimate probability distributions, such as
Monte Carlo dropout [30] or quantile surfaces [31].

Furthermore, we plan to validate our algorithm by incorporating probabilistic
forecasts into an existing planning algorithm [15].

##  V Acknowledgment

This work results from the project DeCoInt2, supported by the German Research
Foundation (DFG) within the priority program SPP 1835: “Kooperativ
interagierende Automobile”, grant numbers DO 1186/1-2 and SI 674/11-2 and the
project KI Data Tooling supported by the Federal Ministry for Economic Affairs
and Energy (BMWi), grant numbers 19A20001L and 19A20001O. Additionally, the
work is supported by “Zentrum Digitalisierung Bayern”.

## References

  * [1] A. R. Ahad, T. Jie, H. Kim, and S. Ishikawa, “Motion history image: Its variants and applications,” _Machine Vision and Applications_ , vol. 23, pp. 255–281, 03 2010.
  * [2] C. G. Keller and D. M. Gavrila, “Will the Pedestrian Cross? A Study on Pedestrian Path Prediction,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 15, no. 2, pp. 494–506, April 2014.
  * [3] R. Quintero, I. Parra, J. Lorenzo, D. Fernandez-Llorca, and M. A. Sotelo, “Pedestrian Intention Recognition by Means of a Hidden Markov Model and Body Language,” in _International Conference on Intelligent Transportation Systems (ITSC)_ , Yokohama, 2017, pp. 1–7.
  * [4] R. Quintero Minguez, I. Parra Alonso, D. Fernández-Llorca, and M. A. Sotelo, “Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 20, no. 5, pp. 1803–1814, May 2019.
  * [5] V. Kress, J. Jung, S. Zernetsch, K. Doll, and B. Sick, “Pose Based Start Intention Detection of Cyclists,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_ , Auckland, New Zealand, Oct 2019, pp. 2381–2386.
  * [6] S. Köhler, M. Goldhammer, S. Bauer, S. Zecha, K. Doll, U. Brunsmann, and K. Dietmayer, “Stationary Detection of the Pedestrian’s Intention at Intersections,” _Intelligent Transportation Systems Magazine, IEEE_ , vol. 5, no. 4, pp. 87–99, 2013.
  * [7] S. Köhler, M. Goldhammer, K. Zindler, K. Doll, and K. Dietmeyer, “Stereo-Vision-Based Pedestrian’s Intention Detection in a Moving Vehicle,” in _2015 IEEE 18th International Conference on Intelligent Transportation Systems_ , Las Palmas, Gran Canaria, Sep. 2015, pp. 2317–2322.
  * [8] S. Zernetsch, V. Kress, B. Sick, and K. Doll, “Early Start Intention Detection of Cyclists Using Motion History Images and a Deep Residual Network,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , Changshu, China, June 2018, pp. 1–6.
  * [9] R. Q. Minguez, I. P. Alonso, D. Fernández-Llorca, and M. A. Sotelo, “Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–12, 2018.
  * [10] M. Goldhammer, S. Köhler, K. Doll, and B. Sick, “Camera based pedestrian path prediction by means of polynomial least-squares approximation and multilayer perceptron neural networks,” in _SAI Intelligent Systems Conference (IntelliSys), 2015_ , London, GB, 2015, pp. 390–399.
  * [11] S. Zernetsch, S. Kohnen, M. Goldhammer, K. Doll, and B. Sick, “Trajectory prediction of cyclists using a physical model and an artificial neural network,” in _2016 IEEE Intelligent Vehicles Symposium (IV)_ , Gothenburg, Sweden, June 2016, pp. 833–838.
  * [12] M. Bieshaar, S. Zernetsch, A. Hubert, B. Sick, and K. Doll, “Cooperative Starting Movement Detection of Cyclists Using Convolutional Neural Networks and a Boosted Stacking Ensemble,” _IEEE Transactions on Intelligent Vehicles_ , vol. 3, no. 4, pp. 534–544, Dec 2018.
  * [13] M. Goldhammer, S. Köhler, S. Zernetsch, K. Doll, B. Sick, and K. Dietmayer, “Intentions of Vulnerable Road Users–Detection and Forecasting by Means of Machine Learning,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–11, 2019.
  * [14] M. Bieshaar, S. Zernetsch, M. Depping, B. Sick, and K. Doll, “Cooperative starting intention detection of cyclists based on smart devices and infrastructure,” in _ITSC_ , Yokohama, Japan, 2017.
  * [15] J. Eilbrecht, M. Bieshaar, S. Zernetsch, K. Doll, B. Sick, and O. Stursberg, “Model-predictive planning for autonomous vehicles anticipating intentions of vulnerable road users by artificial neural networks,” in _2017 IEEE Symposium Series on Computational Intelligence (SSCI)_ , Honolulu, HI, Nov 2017, pp. 1–8.
  * [16] M. Koschi, C. Pek, M. Beikirch, and M. Althoff”, “Set-based prediction of pedestrians in urban environments considering formalized traffic rules,” in _Proc. of the IEEE Int. Conf. on Intelligent Transportation Systems_ , Maui, HI, 2018, pp. 2704–2711.
  * [17] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese, “Social lstm: Human trajectory prediction in crowded spaces,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Las vegas, NV, 2016.
  * [18] E. A. I. Pool, J. F. P. Kooij, and D. M. Gavrila, “Context-based cyclist path prediction using Recurrent Neural Networks,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_ , Paris, France, 2019, pp. 824–830.
  * [19] S. Zernetsch, H. Reichert, V. Kress, K. Doll, and B. Sick, “Trajectory Forecasts with Uncertainties of Vulnerable Road Users by Means of Neural Networks,” in _IV_ , Paris, France, 2019.
  * [20] P. Pinson, H. A. Nielsen, J. K. Møller, H. Madsen, and G. N. Kariniotakis, “Non-parametric probabilistic forecasts of wind power: required properties and evaluation,” _Wind Energy_ , vol. 10, no. 6, pp. 497–516, 2007. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/we.230
  * [21] S. Zernetsch, H. Reichert, S. Schreck, V. Kress, and J. Jung, https://github.com/CooperativeAutomatedTrafficSystemsLab/ProbabilisticCyclistIntentionDetection, 2020\.
  * [22] S. Zernetsch, V. Kress, M. Bieshaar, G. Reitberger, E. Fuchs, K. Doll, and B. Sick, “Cyclist Actions: Motion History Images and Trajectories,” Oct. 2020\. [Online]. Available: https://doi.org/10.5281/zenodo.4080940
  * [23] M. Goldhammer, E. Strigel, D. Meissner, U. Brunsmann, K. Doll, and K. Dietmayer, “Cooperative multi sensor network for traffic safety applications at intersections,” in _2012 15th International IEEE Conference on Intelligent Transportation Systems_ , Anchorage, AK, Sep. 2012, pp. 1178–1183.
  * [24] J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama, and K. Murphy, “Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Honolulu, HI, 2017, pp. 3296–3297.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , Las Vegas, NV, 2016, pp. 770–778.
  * [26] A. Niculescu-Mizil and R. Caruana, “Predicting good probabilities with supervised learning,” 01 2005, pp. 625–632.
  * [27] J. Chambers, _Graphical methods for data analysis_ , ser. Chapman & Hall statistics series. Wadsworth International Group, 1983.
  * [28] K. P. Murphy, _Machine Learning: a Probabilistic Perspective_. Cambridge, MA: MIT Press, 2012.
  * [29] W. J. Morokoff and R. E. Caflisch, “Quasi-monte carlo integration,” _JOURNAL OF COMPUTATIONAL PHYSICS_ , vol. 122, pp. 218–230, 1995.
  * [30] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48_ , ser. ICML’16. JMLR.org, 2016, p. 1050–1059.
  * [31] M. Bieshaar, J. Schreiber, S. Vogt, A. Gensler, and B. Sick, “Quantile surfaces – generalizing quantile regression to multivariate targets,” arXiv: 2010.05898, https://arxiv.org/abs/2010.05898, 2020.

|  Stefan Zernetsch received the B.Eng. and the M.Eng. degree in Electrical
Engineering and Information Technology from the University of Applied Sciences
Aschaffenburg, Germany, in 2012 and 2014, respectively. Currently, he is
working on his PhD thesis in cooperation with the Faculty of Electrical
Engineering and Computer Science of the University of Kassel, Germany. His
research interests include cooperative sensor networks, data fusion, multiple
view geometry, pattern recognition and behavior recognition of traffic
participants.
---|---
|  Hannes Reichert received the B.Eng. and the M.Eng. degree in Electrical
Engineering and Information Technology from the University of Applied Seiences
Aschaffenburg, Germany, in 2018 and 2019, respectively. Currently, he is
working as a research fellow at the University of Applied Seiences
Aschaffenburg, Germany. His research interests include sensor data
abstraction, data augmentation, computer vision, probabilistic modelling and
and uncertainty analysis in machine learning models.
---|---
|  Viktor Kress received the B.Eng. degree in Mechatronics and the M.Eng.
degree in Electrical Engineering and Information Technology from the
University of Applied Sciences Aschaffenburg, Germany, in 2015 and 2016,
respectively. Currently, he is working on his PhD thesis in cooperation with
the Faculty of Electrical Engineering and Computer Science of the University
of Kassel, Germany. His research interests include sensor data fusion, pattern
recognition, machine learning, behavior recognition and trajectory forecasting
of traffic participants.
---|---
|  Konrad Doll received the Diploma (Dipl.-Ing.) degree and the Dr.-Ing.
degree in Electrical Engineering and Information Technology from the Technical
University of Munich, Germany, in 1989 and 1994, respectively. In 1994 he
joined the Semiconductor Products Sector of Motorola, Inc. (now Freescale
Semiconductor, Inc.). In 1997 he was appointed to professor at the University
of Applied Sciences Aschaffenburg in the field of computer science and digital
systems design. His research interests include intelligent systems, their
real-time implementations, and their applications in advanced driver
assistance systems and automated driving. He received several thesis and best
paper awards. Konrad Doll is member of the IEEE.
---|---
|  Bernhard Sick received the diploma, the Ph.D. degree, and the
”Habilitation” degree, all in computer science, from the University of Passau,
Germany, in 1992, 1999, and 2004, respectively. Currently, he is full
Professor for Intelligent Embedded Systems at the Faculty for Electrical
Engineering and Computer Science of the University of Kassel, Germany. There,
he is conducting research in the areas autonomic and organic computing and
technical data analytics with applications in energy systems, automotive
engineering, and others. He authored more than 180 peer-reviewed publications
in these areas. Dr. Sick is associate editor of the IEEE TRANSACTIONS ON
CYBERNETICS. He holds one patent and received several thesis, best paper,
teaching, and inventor awards. He is a member of IEEE (Systems, Man, and
Cybernetics Society, Computer Society, and Computational Intelligence Society)
and GI (Gesellschaft fuer Informatik).
---|---


* Kim et al. [2016] D. Kim, Y. Kwon, P. Liu, I. L. Kim, D. M. Perry, X. Zhang, G. Rodriguez-Rivera,  Apex: Automatic programming assignment error explanation,  in: Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA, volume 02-04-Nove, Association for Computing Machinery, New York, New York, USA, 2016, pp. 311--327. doi:10.1145/2983990.2984031.
  * Yandrapally et al. [2015] R. Yandrapally, G. Sridhara, S. Sinha,  Automated modularization of GUI test cases,  in: Proceedings - International Conference on Software Engineering, volume 1, IEEE Computer Society, New York, New York, USA, 2015, pp. 44--54. doi:10.1109/ICSE.2015.27.
  * Xu et al. [2016] Z. Xu, P. Liu, X. Zhang, B. Xu,  Python predictive analysis for bug detection,  in: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, volume 13-18-Nove, Association for Computing Machinery, New York, New York, USA, 2016, pp. 121--132. doi:10.1145/2950290.2950357.
  * Su et al. [2016] F. H. Su, J. Bell, K. Harvey, S. Sethumadhavan, G. Kaiser, T. Jebara,  Code relatives: Detecting similarly behaving software,  in: Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, volume 13-18-Nove, Association for Computing Machinery, New York, New York, USA, 2016, pp. 702--714. doi:10.1145/2950290.2950321.
  * Clark et al. [2015] S. S. Clark, J. Beal, P. Pal,  Distributed Recovery for Enterprise Services,  in: International Conference on Self-Adaptive and Self-Organizing Systems, SASO, volume 2015-Octob, IEEE Computer Society, New York, New York, USA, 2015, pp. 111--120. doi:10.1109/SASO.2015.19.
  * Brocanelli and Wang [2017] M. Brocanelli, X. Wang,  Smartphone Radio Interface Management for Longer Battery Lifetime,  in: Proceedings - 2017 IEEE International Conference on Autonomic Computing, ICAC 2017, Institute of Electrical and Electronics Engineers Inc., Huddersfield, United Kingdom, 2017, pp. 93--102. doi:10.1109/ICAC.2017.15.
  * Dong et al. [2018] F. Dong, H. Wang, L. Li, Y. Guo, T. F. Bissyandé, T. Liu, G. Xu, J. Klein,  FraudDroid: Automated ad fraud detection for android apps,  in: ESEC/FSE 2018 - Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, Inc, New York, New York, USA, 2018, pp. 257--268. doi:10.1145/3236024.3236045. arXiv:1709.01213.
  * Zhou et al. [2019] X. Zhou, X. Peng, T. Xie, J. Sun, C. Ji, D. Liu, Q. Xiang, C. He,  Latent error prediction and fault localization for microservice applications by learning from system trace logs,  in: ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Association for Computing Machinery, Inc, New York, New York, USA, 2019, pp. 683--694. doi:10.1145/3338906.3338961.
  * Barna et al. [2015] C. Barna, H. Ghanbari, M. Litoiu, M. Shtern,  Hogna: A Platform for Self-Adaptive Applications in Cloud Environments,  in: Proceedings - 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2015, Institute of Electrical and Electronics Engineers Inc., Firenze, Italy, 2015, pp. 83--87. doi:10.1109/SEAMS.2015.26.
  * Casanova et al. [2014] P. Casanova, D. Garlan, B. Schmerl, R. Abreu,  Diagnosing unobserved components in self-adaptive systems,  in: 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2014 - Proceedings, Association for Computing Machinery, New York, New York, USA, 2014, pp. 75--84. doi:10.1145/2593929.2593946.
  * Mertz and Nunes [2017a] J. Mertz, I. Nunes,  Understanding Application-Level Caching in Web Applications,  ACM Computing Surveys 50 (2017a) 1--34.
  * Mertz and Nunes [2017b] J. Mertz, I. Nunes,  A Qualitative Study of Application-Level Caching,  IEEE Transactions on Software Engineering 43 (2017b) 798--816.
  * Mertz et al. [2020] J. Mertz, I. Nunes, L. Della Toffola, M. Selakovic, M. Pradel,  Satisfying increasing performance requirements with caching at the application level,  IEEE Software (2020).
  * Knuth [1997] D. E. Knuth, The Art of Computer Programming, Volume 2 (3rd Ed.): Seminumerical Algorithms, Addison-Wesley Longman Publishing Co., Inc., USA, 1997.