# An Attention-guided Multistream Feature Fusion Network for Localization of
Risky Objects in Driving Videos

Muhammad Monjurul Karim, Ruwen Qin1, Member, Zhaozheng Yin, Senior Member
Muhammad Monjurul Karim, and Ruwen Qin are with the Department of Civil
Engineering, Stony Brook University, Stony Brook, NY 11794, USA.Zhaozheng Yin
is with the Department of Biomedical Informatics, Department of Computer
Science, and AI Institute, Stony Brook University, Stony Brook, NY 11794,
USA.1 Corresponding author: Ruwen Qin, email:
<EMAIL_ADDRESS>received MM DD, YYYY; revised MM DD, YYYY.

###### Abstract

Detecting dangerous traffic agents in videos captured by vehicle-mounted
dashboard cameras (dashcams) is essential to facilitate safe navigation in a
complex environment. Accident-related videos are just a minor portion of the
driving video big data, and the transient pre-accident processes are highly
dynamic and complex. Besides, risky and non-risky traffic agents can be
similar in their appearance. These make risky object localization in the
driving video particularly challenging. To this end, this paper proposes an
attention-guided multistream feature fusion network (AM-Net) to localize
dangerous traffic agents from dashcam videos. Two Gated Recurrent Unit (GRU)
networks use object bounding box and optical flow features extracted from
consecutive video frames to capture spatio-temporal cues for distinguishing
dangerous traffic agents. An attention module coupled with the GRUs learns to
attend to the traffic agents relevant to an accident. Fusing the two streams
of features, AM-Net predicts the riskiness scores of traffic agents in the
video. In supporting this study, the paper also introduces a benchmark dataset
called Risky Object Localization (ROL). The dataset contains spatial,
temporal, and categorical annotations with the accident, object, and scene-
level attributes. The proposed AM-Net achieves a promising performance of
85.73% AUC on the ROL dataset. Meanwhile, the AM-Net outperforms current
state-of-the-art for video anomaly detection by 6.3% AUC on the DoTA dataset.
A thorough ablation study further reveals AM-Net’s merits by evaluating the
contributions of its different components.

###### Index Terms:

accident prediction, risky object detection, autonomous vehicle, attention,
deep learning, dashcam

##  I Introduction

Autonomous driving and Advanced Driver Assistance Systems (ADAS) have
attracted attention and made rapid progress in recent years [1]. Although
research is going in a favorable direction with a beautiful vision of more
comfortable and safer driving experiences, there are still concerns about
traffic accidents. From 2014 to August 23, 2022, 506 autonomous vehicle
collisions were reported in California [2]. Moreover, according to the 2018
Global Status Report on Road Safety from World Health Organization, about 1.35
million people are killed in traffic accidents yearly [3]. Developing
intelligent driving systems to help drivers (or autonomous systems) identify
and localize potential risks is urgently needed to reduce collisions and
fatalities.

Therefore, recognizing and localizing abnormalities in a driving video
captured by vehicle-mounted dashboard cameras (dashcam), one type of low-cost
and widely deployed sensors, is an essential task. Achieving this
functionality can provide a valuable reference for the subsequent path
decision of ADAS. Besides, detecting the presence of a risky object provides
the driver a reference to avoid them. This capability applies to other
applications, such as traffic safety, autonomous driving, and pedestrian
protection [4, 5].

Many computer vision studies have tackled a related task that detects
anomalous events from a dashcam [6, 7, 8, 9, 10, 11]. That is, this stream of
literature focuses on identifying frames where the risk of a traffic accident
is present. Those studies did not localize where in the frame the risk is
present or detect what traffic agents will involve in an accident. Localizing
dangerous objects from a driving scene is critical for ADAS in the real world.

Early prediction and localization of dangerous objects, which are likely to
involve a future accident, from a highly dynamic and complex driving scene
have three crucial challenges. First, the visual appearance of a traffic agent
in the driving scene may not tell much about its risk level because of the
very similar visual appearance of different traffic agents. Second, the time
window for recognizing dangerous objects is short. Lastly, the long-term
temporal dependency of traffic agents underlying the accident risk is hard to
capture. Several attempts have explored this topic, mainly using deep learning
methods [12, 13, 14, 15, 16, 17]. While these studies have laid a solid
foundation for risky object localization, the opportunity to further improve
the performance by extracting rich information from RGB videos is worth an
exploration, especially when the training dataset is limited.

In filling the gap, this paper proposes to build a deep learning framework for
risky object localization from dashcam videos. Several research questions
below motivate the design of the proposed framework. What information, both at
the frame and object levels, can be extracted from the dashcam captured video,
and do they contain essential cues for risky object localization? How are
spatio-temporal patterns of traffic agents learned from the extracted
information, and can the patterns effectively distinguish traffic agents by
their risk level? Should specific traffic agents receive more attention than
others in the driving scene? If that is the case, what attention mechanism to
develop?

The proposed study faces another difficulty. Existing related datasets miss
one or a few pieces of essential information required for developing the
proposed framework to localize dangerous objects in a driving scene.
Therefore, this paper introduces a new dataset, called the Risky Object
Localization (ROL), to address the issue of lacking a suitable dataset. ROL
contains 1,000 accident videos with diverse risky objects in various driving
scenarios. It provides comprehensive annotations at the object, accident, and
scene levels. The annotations in ROL cover the interests of multiple
communities, including intelligent transportation systems, traffic safety,
computer vision, and deep learning. Thus, it not only addresses the need of
this study but will accelerate convergence research like [18] at the interface
of those disciplines.

In summary, this paper makes two contributions by introducing a new attention-
guided multistream network (AM-Net) and a new dataset called ROL. As a result,
the localization of risky traffic agents from the dashcam-captured driving
scene achieves a new state-of-the-art performance. The remainder of this paper
will further detail the discussion by presenting the following contents in
sequence. Section II summarizes the literature to determine the state-of-the-
art. After that, Section III delineates the proposed AM-Net, followed by the
development of the ROL dataset in Section IV. Then, Section V presents
experimental studies to demonstrate and verify the merits of the contributed
network and dataset. In the end, Section VI summarizes research findings and
important future work.

##  II The Literature

This paper is built on studies that contribute to risky object localization,
either directly or indirectly. The related literature is summarized below.

###  II-A Anomaly Detection in a Video

A topic related to risky object localization in a driving scene is video
anomaly detection, which is about finding abnormal events in the video. Video
anomaly detection is often formulated by profiling the normal behavior and
measuring the spatial-temporal feature consistency. Deep learning-based
methods can create a better normal video profile to determine video anomaly by
measuring the consistency. For example, Hasan et al. [19] developed a 3D
Convolutional feed-forward Auto-Encoder (ConvAE) to model regular video
frames. Motivated by this, Chong et al. [20] used a Convolutional Long Short
Term Memory Auto-Encoder (ConvLSTMAE) to simultaneously take advantage of both
Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) in
modeling normal appearance and motion patterns. Liu et al. [16] developed a
Future Frame Prediction (FFP) method to generate future frames using past
observations. At testing, observed deviation against the predicted content
advises for abnormality. In these methods, normal situations are usually
stable scenes, which limits their applicability in driving scenarios where the
camera is in rapid motion.

###  II-B Traffic Accident Anticipation Using Dashcam Videos

Traffic accident anticipation in dashcam videos has become a research focus
recently. Unlike video surveillance systems, dashcam videos capture moving
traffic agents that not only rapidly move but appear and disappear quickly in
the scene. Different advanced methods are developed to learn the spatio-
temporal pattern of the traffic agents to provide an overall riskiness score
of the scene, including an LSTM predictor [6], reinforced learning [8], Graph
Neural Network [7, 11], and dynamic attention [9]. Although they only predict
a risky event in the temporal domain, these studies have developed a solid
methodological foundation for risky object localization in the spatial domain.

###  II-C Risky Object Localization in a Driving Scene

A few studies have attempted to solve the problem of risky object
localization. Ohn-Bar et al. [21] developed a deep spatio-temporal importance
prediction model that assigns riskiness scores to objects in a driving scene.
Kozuka et al. [13] developed a weakly supervised method to predict pedestrian-
involved risky regions. Zeng et al. [14] proposed a soft-attention mechanism
to provide agent-centric riskiness scores for different traffic agents. An RNN
explicitly models non-linear interactions between agents. Although impressive,
their methods did not utilize motion and trajectory features of the input
video sequences, which are critical for risky object localization. Besides,
limiting the number of candidate objects in each frame is not practical in
many driving situations. Li et al. [15] studied the causal impact of risky
objects to the driver’s behavior by removing candidate objects from the input
video stream. While straightforward, it is a highly complex problem to solve
in real-world applications because there could be many known and unknown
casualties. Very recently, Yao et al. [17] proposed to localize risky objects
by predicting future trajectory of traffic participants over a short horizon.
Inconsistent predictions indicate that an anomaly has occurred or is about to
happen. Although achieved a promising result, this method faces a new
challenge arising from objects appearing suddenly in the scene. Those objects
have limited information for predicting their future trajectories reliably.

###  II-D Risky Object Localization Datasets

Several large-scale datasets emerged in response to the growing attention to
deep learning-based risky event anticipation in traffic videos. Chan et al.
[6] created the Dashcam Accident Dataset (DAD) by collecting 620 video clips
of on-road accidents. The last ten frames of each video clip contain an
accident. Yao et al. [22] developed the A3D dataset of 1,500 videos with
annotations of accidents’ starting and ending times. Bao et al. [7] collected
a car accident dataset of 1,500 videos to support traffic accident
anticipation, which also contains annotations of environmental attributes and
accident causes. These datasets are not directly applicable to the problem of
risky object localization due to the lack of object-level risk annotation.
Recently, Yao et al. [17] collected 4,600 videos with the risky object
annotation. They annotated the risky objects contributing to an accident with
their bounding boxes in videos when annotators judge the accident seems
inevitable. This annotation approach is subjective. Besides, their annotations
do not include the beginning time of accidents, which is necessary information
to assess the earliness of a model’s risky object localization ability.

##  III Methodology

In filling the gap in the current studies, this paper proposes an Attention-
Guided Multistream Network (AM-Net) in Fig. 1. The model reads frames of an
input video in their time sequence to predict the riskiness score of any
object in each frame. The model output is the riskiness score of any object
(indexed as $i$) detected in each frame (indexed as $t$), $s_{t,i}$, for
$t=1,2,...$ and any $i$. Traffic accidents usually involve collisions of
different traffic agents. Sudden visual changes in the driving scene, erratic
movement, and chaotic changes in the spatial-temporal relationship between
traffic agents usually indicate that a traffic accident may occur. Therefore,
this study models the risky object localization using multiple features such
as object motion, dynamics of object bounding boxes, and a short history of
the objects in the driving scene. Details of the model are below.

Figure 1: Overview of the proposed AM-Net framework

###  III-A Feature Extraction and Aggregation

AM-Net uses a pretrained object detector YOLOv5 [23] to detect traffic agents
in each frame and provide the bounding boxes of the detected objects in the
frame. $M$ is the number of detected objects, which may vary from one frame to
another. Because the temporal association of the same object in successive
frames is critical information for risky object localization, a multi-object
tracker DeepSort [24] is used to make the association of objects in different
frames.

The change in pixel-level motion among video frames is an important clue to
find objects with unusual movement. Therefore, AM-Net uses a pre-trained RAFT
model [25] to extract the optical flow image for every frame. Then, regions in
the flow image, determined by the detected objects’ bounding boxes, are
cropped to become object-level flow images. Then, the feature extractor
ResNet50 [26] turns the frame-level flow image into a feature vector,
$\boldsymbol{g}_{t}(\in\mathbb{R}^{D})$, and any of those at the object level
into another feature vector, $\boldsymbol{o}_{t,i}(\in\mathbb{R}^{D})$. Here,
$D$ is the dimension of those feature vectors. Then, $\boldsymbol{o}_{t,i}$ is
concatenated with $\boldsymbol{g}_{t}$ to become the overall flow feature
vector for the $i$th object, $\boldsymbol{p}_{t,i}(\in\mathbb{R}^{2D})$:

$\boldsymbol{p}_{t,i}=[\boldsymbol{o}_{t,i};\boldsymbol{g}_{t}],$ (1)

which captures the object’s motion feature in the driving scene. After further
passing a fully connected layer $\phi$, the flow feature vector
$\boldsymbol{p}_{t,i}$ becomes a lower dimensional feature vector,
$\boldsymbol{f}_{t,i}\in(\mathbb{R}^{2d})$:

$\boldsymbol{f}_{t,i}=\phi(\boldsymbol{p}_{t,i};\boldsymbol{\theta}_{0}),$ (2)

where $\boldsymbol{\theta}_{0}$ are learnable parameters of the fully
connected layer.

Bounding boxes’ location and scale changes in successive frames capture the
spatial dynamics of traffic agents over time. Therefore, for any object $i$ in
frame $t$, its bounding box’s location $(x_{t,i},y_{t,i})$ and its scale in
width $w_{t,i}$ and height $h_{t,i}$ are encoded as a feature vector
$\boldsymbol{b}_{t,i}(\in\mathbb{R}^{4})$:

$\boldsymbol{b}_{t,i}=[x_{t,i};y_{t,i};w_{t,i};h_{t,i}].$ (3)

###  III-B Spatio-temporal Relational learning with GRUs

Two Gated Recurrent Units (GRUs) respectively encode the extracted bounding
box feature and flow feature of any detected object into their hidden
representations and update them over time. Fig. 1 shows that, one GRU takes
the bounding box feature vector of object $i$ in frame $t$,
$\boldsymbol{b}_{t,i}$, and the weighted hidden representation of the same
object in the last frame, $\widehat{\boldsymbol{h}}_{b,t-1,i}$, to update its
hidden representation:

$\boldsymbol{h}_{b,t,i}=GRU(\boldsymbol{b}_{t,i},\widehat{\boldsymbol{h}}_{b,t-1,i};\boldsymbol{\theta}_{1}),$
(4)

where $\boldsymbol{\theta}_{1}$ are learnable parameters of the GRU, and
$\boldsymbol{h}_{b,t,i}\in\mathbb{R}^{n}$, where $n$ is the number of hidden
states of the GRU. In parallel, the second GRU takes the flow feature vector
of the object, $\boldsymbol{f}_{t,i}$, and its weighted hidden representation
in the past frame, $\widehat{\boldsymbol{h}}_{f,t-1,i}$, to update the hidden
representation:

$\boldsymbol{h}_{f,t,i}=GRU(\boldsymbol{f}_{t,i},\widehat{\boldsymbol{h}}_{f,t-1,i};\boldsymbol{\theta}_{2}),$
(5)

where $\boldsymbol{\theta}_{2}$ are the learnable parameters, and
$\boldsymbol{h}_{f,t,i}\in\mathbb{R}^{N}$, where $N$ is the number of hidden
states of the GRU. The weighted hidden representations in eqs. (4) and (5)
will be introduced below.

###  III-C Attention Module

Objects in the driving scene have unequal importance for predicting the risk
of traffic accident. Therefore, learnable attentions should be distributed
among the detected objects. Denote $\boldsymbol{H}_{b,t}\in\mathbb{R}^{n\times
M}$ as the hidden representations of the $M$ objects’ bounding box features in
frame $t$:

$\boldsymbol{H}_{b,t}=[\boldsymbol{h}_{b,t,1},\dots,\boldsymbol{h}_{b,t,i},\dots,\boldsymbol{h}_{b,t,M}].$
(6)

Here, the number of detected objects $M$ is a variable that can vary over
time. Since the number of objects is not a fixed number the spatio-temporal
relational learning is not biased to unrelated features.

The dynamic spatial attention weights for the hidden representations of
bounding box features, $\boldsymbol{\alpha}_{b,t}(\in\mathbb{R}^{M})$, are
computed as

$\boldsymbol{\alpha}_{b,t}=\gamma(\tanh(\boldsymbol{H}_{b,t}^{\text{T}})\boldsymbol{w}_{b}),$
(7)

where $\boldsymbol{w}_{b}(\in\mathbb{R}^{n})$ are learnable parameters, and
$\gamma$ represents the softmax operation. Then, $\boldsymbol{\alpha}_{b,t}$
is used to turn $\boldsymbol{H}_{b,t}$ into a weighted aggregation,
$\widehat{\boldsymbol{H}}_{b,t}(\in\mathbb{R}^{n\times M})$:

$\boldsymbol{\hat{H}}_{b,t}=[\widehat{\boldsymbol{h}}_{b,t,1},\dots,\widehat{\boldsymbol{h}}_{b,t,i},\dots,\widehat{\boldsymbol{h}}_{b,t,M}]=\boldsymbol{\alpha}_{b,t}^{\text{T}}\odot\boldsymbol{H}_{b,t}.$
(8)

The same attention mechanism is applied to the hidden representations of flow
features,

$\boldsymbol{H}_{f,t}=[\boldsymbol{h}_{f,t,1},\dots,\boldsymbol{h}_{f,t,i},\dots,\boldsymbol{h}_{f,t,M}].$
(9)

to get their flow feature attention weights
$\boldsymbol{\alpha}_{f,t}(\in\mathbb{R}^{M})$:

$\boldsymbol{\alpha}_{f,t}=\gamma(\tanh(\boldsymbol{H}_{f,t}^{\text{T}})\boldsymbol{w}_{f}),$
(10)

where $\boldsymbol{w}_{f}(\in\mathbb{R}^{N})$ are learnable parameters of the
attention module and $\gamma$ is the softmax operation.
$\boldsymbol{\alpha}_{f,t}$ is applied to $\boldsymbol{H}_{f,t}$ to obtain the
weighted flow hidden representations:

$\boldsymbol{\hat{H}}_{f,t}=[\widehat{\boldsymbol{h}}_{f,t,1},\dots,\widehat{\boldsymbol{h}}_{f,t,i},\dots,\widehat{\boldsymbol{h}}_{f,t,M}]=\boldsymbol{\alpha}_{f,t}^{\text{T}}\odot\boldsymbol{H}_{f,t}.$
(11)

###  III-D Riskiness Score Prediction

As Fig. 1 shows, for any object $i$, the two attention-weighted hidden
representations, $\widehat{\boldsymbol{h}}_{b,t,i}$ and
$\widehat{\boldsymbol{h}}_{f,t,i}$, will respectively flow into the
corresponding GRUs to update the object’s hidden representations in the next
frame $t+1$, as discussed in Section III-B. Meanwhile, they are concatenated
to become the overall hidden representation of the object,
$\widehat{\boldsymbol{h}}_{t,i}(\in\mathbb{R}^{n+N})$:

$\widehat{\boldsymbol{h}}_{t,i}=[\widehat{\boldsymbol{h}}_{b,t,i};\widehat{\boldsymbol{h}}_{f,t,i}].$
(12)

$\widehat{\boldsymbol{h}}_{t,i}$ is decoded by two fully-connected layers
$\phi$ to output the scores of positive and negative classes, which are
further normalized by the softmax operation $\gamma$ to find the riskiness
score of object $i$ in frame $t$, $s_{t,i}$,

${s_{t,i}=\gamma(\phi(\phi(\widehat{\boldsymbol{h}}_{t,i};\boldsymbol{\theta}_{3});\boldsymbol{\theta}_{4})},$
(13)

which is the probability that the object will be involved in an accident soon.

###  III-E The Loss Function

The goal of the model training process is to fit all the learnable parameters
$\boldsymbol{\theta}$’s and $\boldsymbol{w}$’s of the proposed model in
Sections III-A$\sim$III-D by backpropagating a differentiable loss function. A
training dataset contains videos that each has $T$ frames in total and $M$
objects in each frame. In frame $t$, the risk class of the $i$th object, is
indicated by the ground truth label, $l_{t,i}$. $l_{t,i}$ is one if the object
is a risky object (i.e., in the positive class) and zero (i.e., in the
negative class) otherwise. In real-world driving scenarios, positive and
negative classes are imbalanced. Therefore, a weighted cross entropy loss is
calculated for each video in the training dataset:

$\mathcal{L}=-\sum_{t=1}^{T}\sum_{i=1}^{M}\left[w_{p}l_{t,i}\log(s_{t,i})+w_{n}(1-l_{t,i})\log(1-s_{t,i})\right],$
(14)

where $w_{p}$ and $w_{n}$ are the weights for the positive and negative
classes, respectively. Losses of all the training videos are summed up to get
the total loss for optimizing the learnable parameters.

##  IV Dataset Development

This study introduces a new dataset named Risky Object Localization (ROL)
dataset and have made it available to the public [27]. ROL has 1,000 video
clips that each has 100 frames (i.e., 5-seconds long at the frequency 20 fps)
and contains one or multiple risky traffic agents involved in an accident. The
dataset provides the object-level, accident-level, and scene-level annotations
for each video clip. The object-level annotation includes object classes
(positive vs. negative, and traffic agent type), spatial annotation (i.e.,
where the traffic agents are present in frames), and temporal annotation of
risky objects (when a risky object appears). The accident-level annotation
includes accident type (ego involved vs non-involved), manner of collision,
and temporal annotation (when it happens). The scene-level annotation includes
road system, related to roadway, intersection related, weather, and lighting
condition.

TABLE I compares ROL with other traffic accident datasets. DAD, A3D and CCD
datasets do not contain object-level and scene level annotations. Besides,
these three datasets do not contain the manner of collision in their accident
level annotation. DoTA dataset contains annotations at all levels. Yet, it
does not provide the important accident time annotation. Besides, DoTA’s
annotation at the object level are different than the that of the proposed ROL
dataset, as described in Section IV-B.

TABLE I: Comparison between ROL dataset with other datasets. Information of
other datasets are obtained from their released sources.

Dataset | #videos | # frames | fps | Accident | Object | Scene
---|---|---|---|---|---|---
DAD [6] | 620 | 62,000 | 20 | ✓ |  |
A3D [22] | 1,500 | 128,175 | 10 | ✓ |  |
CCD [7] | 1,500 | 75,000 | 10 | ✓ |  |
DoTA [17] | 4,677 | 731,932 | 10 | ✓∗ | ✓⋆ | ✓
ROL (Ours) | 1,000 | 100,000 | 20 | ✓ | ✓ | ✓
$\ast$ accident starting time is not available
$\star$ different than the annotation method of ROL

###  IV-A Data Acquisition

To develop the ROL dataset, videos are collected from the crowd sourced online
platform YouTube using query terms like “traffic accident” and “road crash”. A
long video directly collected from the online platform usually contains many
accidents. Histograms of image pixels in consecutive frames are similar for
the same accident video segment but not for different video segments. Based on
this fact, long videos are segregated into multiple video clips according to
the similarity measurement of frame histograms and using 0.75 as the
threshold. The resulting video clips may contain unnecessary video portions
where the video starts too early before an accident, which have limited
relevance to risky objects or the accident. Similarly, video clips have an
unnecessarily post-accident portion. Therefore, the collected video clips are
trimmed down to 5 seconds each with a frame rate of 20 frames per second
(fps), resulting in 100 frames per video. Resolution of video frames are 1,080
$\times$ 720\. The collected 1000 videos are then randomly divided into 800
training videos and 200 testing videos.

###  IV-B Data Annotation

Fig-2 shows the hierarchy of different annotations in the ROL dataset. This
section further summarizes the temporal, spatial and categorical annotations
of the dataset.

Figure 2: Hierarchy of different annotation in ROL dataset. Blue, green, and
red color indicate categorical, temporal, and spatial annotations,
respectively.

####  IV-B1 Temporal Annotations

Traffic agents in a video that will be involved in the accident in future
frames are considered as risky objects. Both risky objects and the accident
have temporal annotations. Since early detection of accident-involved objects
is essential, this study annotates the time when a risky object appears in the
driving scene for the first time, termed risky object appearance time. Unlike
[17] that annotates the anomaly beginning time as the moment when annotators
think an accident is inevitable, the approach to annotating risky object
appearance time in this paper does not need annotators’ subjective judgment.
The accident beginning time is when a vehicle collides with another static or
dynamic object. Accident beginning time is random in ROL dataset and the
annotation is provided to set the anchor for calculating the earliness of
detecting a risky object.

####  IV-B2 Spatial Annotation

ROL provides bounding boxes as the object-level spatial annotation in a semi-
automated approach that keeps human annotators in the loop. The object
detector YOLOv5 [23] detects traffic agents and renders their bounding boxes.
The multiobject tracker DeepSort [24] further associates the same object over
successive frames as a tracklet and provides a unique tracking ID. By watching
videos overlaid with tracking IDs, human annotators identify the tracking IDs
of risky objects. Solely relying on the object detector and the object tracker
to provide the object-level spatial annotation could miss a few agents, thus
lowering the quality of the dataset. In resolving this problem, the paper
develops a click-to-select object tracker based on DaSiamRPN [28] as an
additional intervention. The tool requires only a few mouse clicks by the
human annotators to select missing or mis-tracked objects. Although it
requires a little additional effort from annotators, the human intervention
ensures the best quality of the dataset. Fig. 3 illustrates six sample frames
in ROL. The first row shows the spatial annotation of risky objects, in the
form of white-shaded bounding boxes. The second row further visualizes object
trajectories, with red curves for risky objects and yellow ones for non-risky
objects. The third row are optical flow images of the sample frames.

Figure 3: Sample images from ROL Dataset

####  IV-B3 Categorical Annotations

Other annotations that ROL provides are about categorical variables,
summarized in TABLE II. Traffic agent class is annotated automatically by
YOLOv5, and others are annotated by human annotators. Annotators also
rechecked the automatically annotated traffic agents and re-annotate miss-
annotated positive traffic agents. Classes of accident-related and scene-
related categorical variables generally follow the definition of FIRST [29].

TABLE II: Summary of categorical variables and distributions

Variable |  Categories and distribution
---|---
Traffic agent class |  i) person (3.0%), ii) car (84.1%), iii) truck (11.2%), iv) bus (0.9%), v) motorcycle (0.5%), vi) bicycle (0.2%)
Risky object class |  i) positive (21.3%), ii) negative (78.7%)
Ego vehicle involvement |  i) ego involved (45.4%), ii) ego non-involved (54.6%)
Road system |  i) local (32.2%), ii) arterial (36.8%), iii) interstate (30.9%)
Related to roadway |  i) roadway (52.5%), ii) shoulder & roadside (12.3%), iii) median (2.4%), iv) intersection related (29.2%), v) others (3.6%)
Intersection related |  i) no intersection (70.8%) ii) 3 way (5.3%), iii) 4 way (20.1%), iv) others (3.8%)
Weather |  i) clear (76.6%), ii) cloudy (12.9%), iii) rain (8.5%), iv) snow (3.9%)
Lighting condition |  i) day (81.3%), ii) night (18.7%)
Manner of collision |  i) angle (36.4%), ii) sideswipe (11.6%), iii) rear-end (19.2%), iv) head-on (6.0%), v) others (26.8%)

##  V Results and Discussion

Experiments are conducted to verify the effectiveness of the proposed AM-Net
and the newly developed ROL dataset. Implementation details, evaluation
metrics, and the results are discussed below.

###  V-A Implementation Details

The proposed AM-Net is built using PyTorch [30]. Model training and testing
are performed using an Nvidia Tesla V100 GPU with 32GB of memory. All the
input frames are resized to 224$\times$224 before feeding to the ResNet50 [26]
feature extractor. Feature vectors $\boldsymbol{g}_{t,i}$ and
$\boldsymbol{o}_{t,i}$ are obtained by applying an average pooling operation
to the output of the ResNet50 feature extractor, and their dimension ($D$) is
2,048. A fully-connected layer further reduces the dimension to 256 ($d$). The
dimension of flow hidden representations ($N$) is 256 and that for bounding
box hidden representations ($n$) is 32. A learning rate of 0.001 is used to
train the AM-Net on the newly developed ROL dataset, and ReduceLROnPlateau is
used as the learning rate scheduler. Adam optimizer is used to optimize the
network for 30 epochs and the best model is selected. The positive to negative
class ratio in ROL is $0.27:1$. Therefore, the class weights for the negative
class ($w_{n}$) and positive class ($w_{p}$) are selected as 0.27 and 1,
respectively.

###  V-B Evaluation Metrics

The evaluation of model performance focuses on two aspects: the correctness in
localizing risky objects in videos and the earliness of risky object
detection. To evaluate the correctness, this study uses Area under the
Receiver Operating Characteristic Curve (AUC). AUC can measure the ability of
AM-Net to differentiate risky and non-risky traffic agents.

To measure the earliness of the prediction, mean Time-to-Accident (mTTA) is
used. Time-to-Accident (TTA) is defined as the first time when a riskiness
score $s_{t,i}$ goes across a threshold value $\bar{s}$. That is,

$\text{TTA}=\max\\{\tau-t|s_{t,i}\geq\bar{s},0\leq t\leq\tau,\forall\,i\\},$
(15)

where $\tau$ is the accident beginning time. TTA is dependent of the selection
of a threshold value $\bar{s}$. mTTA, the average of TTA values at different
threshold values, is calculated as an earliness metrics independent of the
threshold value selection.

###  V-C Evaluation of the Proposed Model Architecture

An ablation study is conducted to evaluate the contribution of different
components of AM-Net, which compares the correctness and earliness metrics of
AM-Net and nine variants of it. Those ten models are trained and tested on
ROL, and results are summarized in TABLE III.

TABLE III: Ablation study on the ROL dataset.

Model | RGB | Bbox | Flow | Att. | AUC | mTTA
---|---|---|---|---|---|---
O | F | O | F | (%) | (s)
1 | ✓ | ✓ |  |  |  |  | 80.13 | 2.30
2 |  |  | ✓ |  |  |  | 82.01 | 2.24
3 |  |  |  | ✓ | ✓ |  | 81.49 | 2.09
4 | ✓ | ✓ | ✓ |  |  |  | 83.46 | 2.16
5 | ✓ | ✓ |  | ✓ | ✓ |  | 82.43 | 2.05
6 | ✓ | ✓ | ✓ | ✓ | ✓ |  | 85.11 | 2.08
7 | ✓ |  | ✓ | ✓ | ✓ |  | 85.20 | 2.01
8 |  |  | ✓ | ✓ | ✓ |  | 85.47 | 2.05
9 |  |  | ✓ | ✓ |  |  | 84.11 | 1.87
10 |  |  | ✓ | ✓ | ✓ | ✓ | 85.73 | 2.34
O: object level feature; F: frame level feature; Att: Attention;
Bbox: Bounding box feature

Model #1 from the table solely uses the object-level and frame-level
appearance features that can be directly extracted from the RGB images as the
input for risky object localization. This model achieves 80.13% AUC with 2.30
second mTTA. This performance sets a benchmark for evaluating the proposed AM-
Net (i.e., model #10) and its variants (models #2$\sim$9). Model #2 uses the
bounding box feature as the single input, which increases AUC to 82.13% but
reduces mTTA slightly for 0.06 second. This suggests that the bounding box
feature is better than appearance features to improve the correctness of
detecting risky objects in traffic videos. Changes in bounding box location
and scales are more discriminative than appearance features for risky object
detection. Similarly, Model #3 uses flow features as the input. The comparison
between models #1 and #3 shows flow features in standalone can improve AUC for
1.36%, but it reduces mTTA for 0.21 second, in detecting risky objects. In
short, the bounding box feature and flow features can respectively improve the
correctness, but lower the earliness, in predicting risky objects.

The fusion of bounding box feature with appearance features (model #4)
achieves a higher AUC than using any individual feature type (models #1 and
#2), but not longer mTTA. The similar conclusion can be made by comparing the
fusion of flow features with appearance features (model #5) to the use of
individual feature type (models #1 and #2). Fusing those three types of
features together (model #6) effectively brings AUC to 85.11%. Models
#4$\sim$#6 confirms the effectiveness of feature fusion.

Model #7 removes the frame-level appearance feature input from model #6, which
increases the AUC slightly, for 0.09%, but sacrifices the mTTA for 0.07
second. Model#8 removes both the frame-level and object-level appearance
features from model #6, resulting an additional increase in AUC. Model #9
removes the frame-level flow feature of model #8, resulting in lower AUC and
shorter mTTA. The frame-level flow feature captures the motion of the traffic
scene due to the ego-vehicle movement and the motion of other objects in the
same frame, which are important clue for identifying risky objects in the
frame. The comparison of those four models (#6$\sim$#9) suggests the fusion of
bounding box feature and flow features (both the object level and the frame
level) is the most suitable design of feature fusion. Appearance features,
while are effective in other tasks, may lower the ability to differentiate
objects by their riskiness classes.

Finally, model #10 (i.e., the proposed AM-Net) adds the proposed attention
module to model #8, which achieves the best performance, 85.73% AUC and 2.34
second mTTA. The attention module not only effectively addresses the
limitation of bounding box feature and flow features in detecting risky
objects earlier but further increases the correctness.

###  V-D Comparison to the State-of-the-Art Models

This study compares the AM-Net (both without and with the attention module)
trained on ROL (i.e., models #8 and #10 in TABLE III) to existing models [19,
20, 16, 17] on the testing dataset of DoTA [17] from the perspective of video
anomaly detection. That is, they are compared on the metric of frame-level
AUC. The earliness of detection is skipped in this comparison because DoTA
testing dataset does not have the annotation of accident beginning time. TABLE
IV summarizes results of the comparative study. The frame-level AUC values of
models [19, 20, 16, 17] are provided by [17]. This paper computes the frame-
level AUC for the proposed AM-Net. To compute the frame-level AUC with the
output of AM-Net, the highest riskiness score of any detected objects in a
frame, $\max_{i}\\{s_{t,i}\\}$, is considered as the riskiness score for the
frame. A frame containing a risky object is a positive sample. The Receiver
Operating Characteristic (ROC) curve is attained by calculating the true
positive rate and false positive rate at various detecting threshold values.
Then, the frame-level AUC is calculated accordingly.

TABLE IV: Comparison of the proposed model with existing methods on DoTA [17]
testing dataset for video anomaly detection

Method | Features | Frame-level AUC(%)
---|---|---
ConvAE [19] | Flow (F) | 66.3
ConvLSTMAE [20] | Flow (F) | 62.5
FFP [16] | RGB(F) | 67.5
FOL [17] | RGB(F) + Bbox + Flow(O) + Ego | 73.0
AM-Net (Ours) | Bbox + Flow (O+F) | 79.3
| Bbox + Flow (O+F) + Attn. | 75.7

TABLE IV shows that ConvAE and ConvLSTMAE with the frame-level flow feature as
the only input achieve 66.3% and 62.5% AUC, respectively. FFP, which uses the
frame-level appearance feature, achieves 67.5% AUC. FOL effectively boosts up
the performance to 73.0% by fusing the bounding box, ego motion, and object-
level optical flow feature with the frame-level appearance feature. The AM-Net
without the attention module achieved 79.3%, about 6.3% higher than FOL
attains. The improvement of AM-Net above FOL can be associated with the choice
of features and the method that the video anomaly detection is based upon. The
comparison also signifies that the bounding box and flow features are the most
important features to capture the spatio-temporal pattern of risky objects.
When testing the AM-Net with the attention module on the DoTA testing dataset,
a 75.71% frame-level AUC is achieved. Adding the attention module to AM-Net
does not attain the best performance in this test because AM-Net is trained on
ROL that has 20 fps but tested on DoTA that has only 10 fps. The temporal
dynamics of the spatial relationship between different objects are hampered
when a large portion of a video sequence is skipped. Therefore, when the frame
frequency of testing videos is low, the spatial attention learnt from a
training dataset in higher frame frequency has a negative effect. It should be
noted that the performance of AM-Net can be increased further if it was both
trained and tested on the DoTA dataset.

###  V-E Performance by Manner of Collision

The spatio-temporal pattern of risky objects varies with the manner of
collision. To reveal how well the proposed AM-Net is performing in localizing
risky objects involved in different manners of collision, this study
calculates the AUC and mTTA by the manner of collision. The testing dataset of
ROL contains 34.0% angle, 18.5% sideswipe, 10.0% rear-end, 5.5% head-on, and
32.0% other types of collision videos.

TABLE V: Performance by manner of collision Manner of collision | AUC (%) | mTTA (s)
---|---|---
Angle | 84.71 | 2.31
Sideswipe | 88.95 | 2.13
Rear-end | 90.00 | 1.97
Head-on | 76.32 | 2.14
Others | 82.03 | 2.70
Figure 4: Examples of risky object localization.

The highest AUC that AM-Net achieves is 90.00%, in predicting risky objects in
the rear-end collision. Risky objects in this manner of collision are in front
and seem less challenging to localize. While localizing risky objects in the
sideswipe collision is more difficult than in the rear-end collision, AM-Net
still attains 88.95% AUC. The AUC pertaining to the angle collision is 84.71%,
and the large variability of videos with this manner of collision is one
possible reason that correctly localizing risky objects becomes more
difficult. AM-Net only achieves 76.32% AUC for the head-on collision. Data for
head-on collision is very rare in natural driving videos. Hence, the small
portion of training videos in this collision type is the major reason for the
relatively weaker performance. On testing videos of other manners of
collision, AM-Net achieves 82.03% AUC, mainly due to the large variability of
the spatio-temporal pattern of risky-objects.

If other manners of collisions are not counted, AM-Net is the fastest in
localizing risky objects in the angle collision, using 2.31 second on average.
AM-Net can predict risky objects in the sideswipe and head-on collision in
about 2.13$\sim$2.14 second, but slightly less than 2 second in the rear-end
collision. More or less, the relatively short mTTA in localizing risky objects
in the rear-end collision pertains to the relatively close distance of those
objects to the ego-vehicle in this manner of collision.

The experimentation with respect to different manners of collision confirms
that AM-Net can achieve promising performance in localizing risky objects,
particularly in angle, sideswipe, and rear-end collision types. As per NHTSA
report [29], 66% of reported collisions with body injury or property damage
pertain to these three manners of collision. Correctly localizing risky
objects ahead of time would allow for taking preventive actions to avoid the
accident.

###  V-F Qualitative Evaluation

Fig. 4 illustrates three examples in the ROL dataset to demonstrate the risky
object localization result of AM-Net. In each example, the true risky objects
are highlighted by white-shaded bounding boxes. The riskiness score of true
risky objects in a frame is one as long as it appears in that frame, which is
indicated by the green curve in the bottom row. Risky objects localized by AM-
Net are overlaid with colored bounding boxes. For the illustration purpose, a
threshold value of 0.3 is set to determine if a predicted object is a risky
object. Therefore, the color scheme for bounding boxes shown in Fig. 4, which
is a reddish black color to a yellowish white color, is mapped to the range
[0.3, 1]. The riskiness score of riskiness objects are presented as curves in
colors other than green.

The video of example (a) contains two risky objects, which are two vehicles
far away from the ego vehicle. They appear in the video from the beginning
and, thus, the ground truth riskiness score (i.e., the green line) is one
throughout the timeline of the video sequence. The video contains a risky
situation where two vehicles are approaching each other from the side,
resulting in a collision starting at frame #70. The accident scene is complex
because the two risky objects are very small due to the far distance from the
ego-vehicle. AM-Net successfully assigns the highest riskiness scores to those
two objects from a very early stage. The color of the first risky object’s
bounding box (in the middle row of the example) quickly turns from red to
yellow, and the riskiness score (the red curve in the bottom row of the
example) increases gradually. For the second risky object, its riskiness score
(the blue curve) is a little lower than that of the first object. It is mainly
because of the relatively lower velocity and relatively smaller size of the
second vehicle than the first one. After reaching the 0.3 threshold value, the
network successfully maintains the riskiness score above the threshold value
across the timeline.

The video of example (b) is an ego vehicle-involved accident. One vehicle (the
risky object) cuts into the direct lane of the ego vehicle from the side,
resulting in a collision at frame #36. In this example, the risky object
appears in the video from the beginning. AM-Net assigns the highest riskiness
score to the correct vehicle in the traffic.

In the video of example (c), the risky object is not there until frame #33
when the vehicle is coming to the front of the ego vehicle from the opposite
direction, resulting in an angle collision at frame #70. Although the risky
object appears for just a short period, AM-Net does not predict any false
positive detection. Instead, it successfully assigns the highest riskiness
score to the correct vehicle.

##  VI Conclusion

This paper presented a novel deep learning framework named AM-Net for
localizing risky traffic agents from dashcam videos. AM-Net extracts the
spatial information (i.e., bounding boxes) and motion information (i.e.,
optical flows) of traffic agents in the driving scene and learns those agents’
spatio-temporal features from such information in successive frames. The
network effectively localizes risky agents in the driving videos by fusing the
multistream features at the object and frame levels and learning to
differentiate its attention to different agents. A thorough ablation study not
only justifies the rationale for selecting bounding boxes and optical flows as
the input but verifies the effectiveness of the proposed feature fusion and
attention mechanism. Moreover, experimental evaluation confirms that AM-Net
has exceeded the state-of-the-art. The paper also introduced ROL, a benchmark
dataset containing object-level, accident-level, and scene-level annotations
for the task of risky object localization in dashcam videos. Besides, the rich
annotations of ROL capture multiple communities’ interests and can fuel
multidisciplinary research on intelligent systems for transportation safety
enhancement.

This research builds a foundation for exploring advanced opportunities for
improving the performance of risky object localization, such as multimodal
sensor fusion, multi-camera perception, and Vehicular Ad-hoc Networks (VANET).
For example, camera-LiDAR fusion allows for extending this paper by building
the model upon reliable 3D object detection and tracking. 3D visual perception
based on a multi-camera system has become a critical task of autonomous
driving. Localizing dangerous traffic agents 360∘ around the ego vehicle will
vastly improve safety. Integrating the AM-Net into the Vehicular Ad hoc
Network (VANET) environment allows for relaying the information of risky
traffic agents to connected vehicles and infrastructure to minimize the impact
of a potential accident on the traffic.

While AM-Net has achieved promising performance in localizing risky traffic
agents and anticipating traffic accidents, its limitations are present and
should be addressed in future research. Data scarcity limits the early
localization of the risky objects from head-on collisions in this study.
Incorporating the prior knowledge from the dataset into the model learning to
distribute more attention to the front-facing vehicles approaching the ego
vehicle may address this issue. Another limitation is that this study
considers only six types of traffic agents. The model can further include
other traffic agents, especially animals in specific areas and road debris.

## Acknowledgment

It will be added after the manuscript is accepted for publication.

## References

  * [1] A. Eskandarian, C. Wu, and C. Sun, “Research advances and challenges of autonomous and connected ground vehicles,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 2, pp. 683–711, 2021.
  * [2] California DMV, “Autonomous vehicle collision reports,” 2022, available at:https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-vehicles/autonomous-vehicle-collision-reports/, accessed September, 2022.
  * [3] WHO, “Global status report on road safety 2018: Summary,” World Health Organization, Tech. Rep., 2018.
  * [4] U. Alvi, M. A. K. Khattak, B. Shabir, A. W. Malik, and S. R. Muhammad, “A comprehensive study on iot based accident detection systems for smart vehicles,” _IEEE Access_ , vol. 8, pp. 122 480–122 497, 2020.
  * [5] K. Muhammad, A. Ullah, J. Lloret, J. Del Ser, and V. H. C. de Albuquerque, “Deep learning for safe autonomous driving: Current challenges and future directions,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 7, pp. 4316–4336, 2020.
  * [6] F.-H. Chan, Y.-T. Chen, Y. Xiang, and M. Sun, “Anticipating accidents in dashcam videos,” in _Asian Conference on Computer Vision_. Springer, 2016, pp. 136–153.
  * [7] W. Bao, Q. Yu, and Y. Kong, “Uncertainty-based traffic accident anticipation with spatio-temporal relational learning,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 2682–2690.
  * [8] ——, “DRIVE: Deep reinforced accident anticipation with visual explanation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7619–7628.
  * [9] M. M. Karim, Y. Li, R. Qin, and Z. Yin, “A dynamic spatial-temporal attention network for early anticipation of traffic accidents,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 9590–9600, 2022.
  * [10] M. M. Karim, Y. Li, and R. Qin, “Toward explainable artificial intelligence for early anticipation of traffic accidents,” _Transportation Research Record_ , vol. 2676, no. 6, pp. 743–755, 2022.
  * [11] J. Fang, J. Qiao, J. Bai, H. Yu, and J. Xue, “Traffic accident detection via self-supervised consistency learning in driving scenarios,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 7, pp. 9601–9614.
  * [12] M. Gao, A. Tawari, and S. Martin, “Goal-oriented object importance estimation in on-road driving videos,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 5509–5515.
  * [13] K. Kozuka and J. Carlos Niebles, “Risky region localization with point supervision,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2017, pp. 246–253.
  * [14] K.-H. Zeng, S.-H. Chou, F.-H. Chan, J. Carlos Niebles, and M. Sun, “Agent-centric risk assessment: Accident anticipation and risky region localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2222–2230.
  * [15] C. Li, S. H. Chan, and Y.-T. Chen, “Who make drivers stop? towards driver-centric risk assessment: Risk object identification via causal inference,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 10 711–10 718.
  * [16] W. Liu, W. Luo, D. Lian, and S. Gao, “Future frame prediction for anomaly detection–a new baseline,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6536–6545.
  * [17] Y. Yao, X. Wang, M. Xu, Z. Pu, Y. Wang, E. Atkins, and D. Crandall, “DoTA: Unsupervised detection of traffic anomaly in driving videos,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2022.
  * [18] Y. Li, M. M. Karim, R. Qin, Z. Sun, Z. Wang, and Z. Yin, “Crash report data analysis for creating scenario-wise, spatio-temporal attention guidance to support computer vision-based perception of fatal crash risks,” _Accident Analysis & Prevention_, vol. 151, p. 105962, 2021.
  * [19] M. Hasan, J. Choi, J. Neumann, A. K. Roy-Chowdhury, and L. S. Davis, “Learning temporal regularity in video sequences,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 733–742.
  * [20] Y. S. Chong and Y. H. Tay, “Abnormal event detection in videos using spatiotemporal autoencoder,” in _International Symposium on Neural Networks_. Springer, 2017, pp. 189–196.
  * [21] E. Ohn-Bar and M. M. Trivedi, “Are all objects equal? deep spatio-temporal importance prediction in driving videos,” _Pattern Recognition_ , vol. 64, pp. 425–436, 2017.
  * [22] Y. Yao, M. Xu, C. Choi, D. J. Crandall, E. M. Atkins, and B. Dariush, “Egocentric vision-based future vehicle localization for intelligent driving assistance systems,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 9711–9717.
  * [23] Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “YOLOX: Exceeding YOLO series in 2021,” _arXiv preprint arXiv:2107.08430_ , 2021.
  * [24] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in _2017 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2017, pp. 3645–3649.
  * [25] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in _European Conference on Computer Vision_. Springer, 2020, pp. 402–419.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [27] M. M. Karim, D. Racz, G. Liu, Z. Li, Y. Gong, W. Li, M. Incardona, R. Qin, and Z. Yin, “Risky object localization (ROL) in a driving scene dataset,” 2022\. [Online]. Available: https://github.com/monjurulkarim/ROL_Dataset
  * [28] Z. Zhu, Q. Wang, B. Li, W. Wu, J. Yan, and W. Hu, “Distractor-aware siamese networks for visual object tracking,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 101–117.
  * [29] NHTSA, “Fatality and injury reporting system tool (FIRST),” 2022, available at:https://cdan.nhtsa.gov/query, accessed July, 2022.
  * [30] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in Neural Information Processing Systems_ , vol. 32, pp. 8026–8037, 2019.