# Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review

Abofazl Razi ∗222These authors contributed equally to this work., Xiwen
Chen222These authors contributed equally to this work., Hao Wang,
School of Computing
Clemson University
&Huayu Li
Department of Electrical and Computer Engineering
University of Arizona
&Brendan Russo
Department of Civil Engineering
Northern Arizona University
&Yan Chen
The Polytechnic School
Arizona State University &Hongbin Yu
School of Electrical, Computer and Energy Engineering
Arizona State University

###### Abstract

This paper explores Deep Learning (DL) methods that are used or have the
potential to be used for traffic video analysis, emphasizing driving safety
for both Autonomous Vehicles (AVs) and human-operated vehicles. We present a
typical processing pipeline, which can be used to understand and interpret
traffic videos by extracting operational safety metrics and providing general
hints and guidelines to improve traffic safety. This processing framework
includes several steps, including video enhancement, video stabilization,
semantic and incident segmentation, object detection and classification,
trajectory extraction, speed estimation, event analysis, modeling and anomaly
detection. Our main goal is to guide traffic analysts to develop their own
custom-built processing frameworks by selecting the best choices for each step
and offering new designs for the lacking modules by providing a comparative
analysis of the most successful conventional and DL-based algorithms proposed
for each step. We also review existing open-source tools and public datasets
that can help train DL models. To be more specific, we review exemplary
traffic problems and mentioned requires steps for each problem. Besides, we
investigate connections to the closely related research areas of drivers’
cognition evaluation, Crowd-sourcing-based monitoring systems, Edge Computing
in roadside infrastructures, Automated Driving Systems (ADS)-equipped
vehicles, and highlight the missing gaps. Finally, we review commercial
implementations of traffic monitoring systems, their future outlook, and open
problems and remaining challenges for widespread use of such systems333This
material is based upon the work supported by the National Science Foundation
under Grant No. 2008784 and the Arizona Commerce Authority under the Institute
of Automated Mobility (IAM) project..

11footnotetext: Corresponding author. email<EMAIL_ADDRESS>
##  1 Introduction

Despite recent advances in vehicles’ operational safety features, computer-
assisted control, and technology-based traffic management systems, traffic
safety remains one of the main challenges in today’s life. Every year, traffic
crashes account for 20-50 million causalities and 1.35 million fatalities
worldwide, making it one of the top-10 causes of death. Indeed, traffic
crashes is the leading cause of death for people aged 5 to 29 [1].

Figure 1: A framework of Computer Vision (CV)-based traffic safety analysis
pipeline.

Figure 2: The data sources and computing resources used in video-based traffic
analysis.

Using computational intelligence and computer tools for enhancing traffic
safety has gained a lot of attention in recent years. Mainstream technological
trends include (i) implementing vehicle safety features such as forward
collision warning, blind-spot detection, lane departure warning, backup
camera, and autonomous emergency braking [2], (ii) simulation-based road
infrastructure design such as Site3D, RoadEng [3], and OpenRoads Designer [4],
and (iii) intelligent traffic flow management systems such as Global
Positioning System (GPS)-based navigation tools. An example of the last
category is Google’s road-user interpretive software that can infer the common
road behavior of other drivers that allows Engine Control Units (ECUs) to make
better route decisions [5].

Many car manufacturers have developed Artificial Intelligence (AI)-based
safety features [5]. Tesla, Audi, and BWM have already developed sensor-based
and vision-based perception systems to help the drivers judge road conditions
more accurately and use partial autonomy [6, 7]. Particularly, the recent
advances in Deep Learning (DL) methods for video processing backed by low-cost
and high-speed computational platforms such as Graphics/Tensor Processing
Units (GPUs/TPUs) have accelerated the pace of developing AI-based features
both at vehicle and infrastructure levels [8].

Commercial transportation has witnessed unprecedented growth in utilizing
autonomous driving systems, in recent years. Waymo launched a self-driving
taxi service in Phoenix, Arizona in 2020 [9]. Almost the same time, General
Motors (GM) presented their prototype for autonomous buses [10]. Meanwhile,
Amazon revealed their future public transportation plan, which offers
passengers-only compact autonomous shuttles [11]. Starship robots [12] and
Nuro’s autonomous delivery bots [13] are other examples of AI-based commercial
traffic platforms. The autonomous semi-truck concept, independently presented
by Tesla and Waymo [14, 15], brings tons of benefits to logistic and cold
chain monitoring, building a solid fundamental for industry traffic.

In the industry domain, camera-based traffic monitoring systems have been
broadly used to detect incidents and apply congestion control [16, 17].
Sensor-based adaptive traffic light control systems are used to reduce traffic
jams [18, 19, 20]. Modern systems use Vehicle-To-Vehicle (V2V) and Vehicle-To-
Infrastructure (V2I) communications to enhance the overall driving safety. For
instance, Roadside Unit (RSU)-based radar systems with real-time data analysis
are used to monitor pedestrians on the road and minimize crash rates by
sending alerts to connected vehicles [21]. These technologies have optimized
existing infrastructures and greatly increased traffic safety.

###  1.1 Architectural View

Traffic Safety analysis can be viewed as a modular and multi-faceted problem
that involves many aspects. As shown in Figs. 1 and 2, the overall analysis
platform can be viewed as a software pipeline where the collected information
undergoes different processing steps until it is translated to navigation
commands, advisory messages, or overall guidelines for improving traffic
safety.

This paper provides a comprehensive review of the popular methods, tools,
software packages, and datasets developed by the scientific community for each
vision-based sub-problem. We also highlight the open problems and future
challenges on each frontier. Our main emphasis is on the role of vision-based
DL methods in enhancing traffic flow (e.g., improving efficiency) and
mitigating traffic safety risks.

In contrast to the primary trend of processing individual events from an
involved vehicle’s perspective, we consider traffic safety at both the
vehicle-level and network-level by processing videos captured by an external
observer. The main information source is captured video by vehicle-mounted
cameras and roadside cameras, but we also will review other sensor information
that can be used for traffic management. We believe that enabling advanced
traffic safety analysis and monitoring platforms will play a crucial role in
future smart cities. Fig.3 compares video-analysis from two different
perspectives, from an external point of view as well as an internal node’s
perspective.

Table 1: Summary of related review papers. The paper with ’*’ means although
this paper is related but outdated. $L\surd$ denotes a topic is covered in
fewer details.

Paper |  [22]Hu et al. |  [23]Mozaffari et al. |  [24]Grigorescu et al. |  [25]Yurtsever et al. |  [26]Janai et al. |  [27]Badue et al. |  [28]kumaran et al. |  [29]Wang et al. |  [30]Nguyen et al. | *[31]shirazi et al. | *[32]Mukhtar et al. | *[33] Morris et al. | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Year | 2020 | 2020 | 2020 | 2020 | 2020 | 2020 | 2019 | 2019 | 2018 | 2016 | 2015 | 2013 |
| Human-driven
---
Vehicle
$\surd$ | $\surd$ |  | $\surd$ |  |  | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
AVs | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ |  | $L\surd$ |  |  |  | $\surd$
| Safety Assessment
---
Analysis
$\surd$ | $\surd$ | $\surd$ | $\surd$ |  |  | $\surd$ |  |  | $\surd$ |  | $\surd$ | $\surd$
| CV-based
---
Method
$\surd$ | $L\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $L\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$
| Deep Learning
---
Method
$L\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $L\surd$ | $\surd$ | $\surd$ | $L\surd$ |  |  |  | $\surd$
Sensors | $\surd$ |  | $\surd$ | $\surd$ | $\surd$ | $\surd$ |  |  |  | $\surd$ | $\surd$ |  | $\surd$
Datasets |  |  | $\surd$ | $\surd$ | $\surd$ |  | $\surd$ |  |  | $\surd$ |  | $\surd$ | $\surd$
| Network
---
Analysis
$\surd$ | $\surd$ |  |  |  |  | $\surd$ | $\surd$ | $\surd$ |  |  | $\surd$ | $\surd$
| Vehicular
---
Edge Computing
|  |  |  |  |  |  |  |  |  |  |  | $\surd$
| Behavioral &
---
Driver Cognition
|  |  |  |  |  |  |  |  |  |  |  | $\surd$

Figure 3: The example of Perspectives: (a) Internal Perspective and (b)
External Perspective. Table 2: Specific details on the shortcomings of the
most recent survey papers in CV-based traffic analysis.

Survey | Content | Drawback | Perspective | Application
---|---|---|---|---
[22](2020) |  | Review existing research works on traffic conflicts based on perception
---
technologies and communication technologies.
| Discusses very limited DL methods (covers less than 10 DL methods);
---
covers in fewer details the CV tasks
(including detection, trajectory extraction, and video analysis).
Internal | AVs
[23](2020) |  | Reviews DL approaches for Trajectory prediction based on
---
the input representation, output type, and prediction method.
| Because of the limited works focusing on CV data (e.g. BEV),
---
it only discusses 6 (CNN)+4 (CNN+RNN) methods.
External | Avs; traffic analysis
[24](2020) |  | Reviews DL approaches (mainly as CNN, RNN,
---
Deep Reinforcement Learning) for AVs;
partially discusses safety assessment.
| shallowly covers sensor information;
---
less details are provided for performance comparison
(only covers 10 detection methods on a general dataset and
4 semantic segmentation methods on CityScapes).
Internal | AVs
[25](2020) |  | Reviews practical challenges and solutions for Avs;
---
the main tasks involve localization, mapping,
perception, planning, and human-machine interfaces;
emphasizes the whole driving system architecture.
| Excludes video pre-processing methods (e.g., Video stabilization);
---
provides little details on the performance analysis
(9 methods on ImageNet and 7 methods on KITTI 3D).
Internal | AVs
[26](2020) |  | Reviews CV-based approaches for AVs;
---
discusses open problems and current challenges.
| Lacks the safety assessment analysis;
---
does not give detailed performance comparisons.
Internal | AVs
[27](2020) |  | Reviews technologies for the perception and
---
decision-making system of Avs;
discusses the AV industry milestones.
| Covers limited DL methods (sign and light detection,
---
pavement marking detection,
MOT, excluding segmentation methods);
does not give detailed performance comparisons.
Internal | AVs
[28](2019) |  | Reviews approaches for
---
anomaly detection for surveillance videos.
| Only includes tasks related to anomaly detection;
---
does not include other CV tasks;
does not provide comprehensive performance analysis.
External | Traffic analysis
[29](2019) |  | Reviews DL approaches for multiple traffic applications,
---
including time series prediction,
classification, and optimization.
| Covers a few CV-based problems (including traffic sign recognition,
---
vehicle detection, and pedestrian detection,
excluding MOT and segmentation);
offers a limitted comparative analysis
(including only the traffic sign and traffic flow prediction).
External | AVs; traffic analysis
[30](2018) | Reviews DL approaches for processing traffic data. |  | Covers a few CV-based problems (including vehicle detection,
---
perception on Avs, etc,.) without sufficient detail.
External | Traffic analysis

It is noteworthy that several review papers have been published to review
methods and tools used for video-based traffic analysis. A summary of these
papers is provided in Table 1. However, most review papers have limitations in
certain aspects.

Some survey papers (e.g., [25, 26, 27, 29, 30]) focused merely on solving
traffic-related tasks (such as perception) while not covering safety
assessment methods. Some other papers (e.g., [22, 27]) do not provide a
comprehensive summary of DL methods, which recently has become the dominant
approach in both industry and academia research. Other papers (e.g., [25, 26,
27]), which review driving techniques for vehicles equipped with Automated
Driving Systems (ADS), keep their attention solely on the DL methods developed
for AVs while not investigating the practicality of these methods on human-
driven vehicles, which still are the most widely used vehicles. A few papers
(e.g., [28, 22]) put their primary focus on DL methods but without special
emphasis on vision-related tasks particularly useful for traffic analysis. A
different set of papers (e.g., [23]) limit their analysis to the perspective
of internal nodes, also known as the first-person perspective. Although
Computer Vision (CV)-based tasks are covered by several papers [29, 30], they
do not provide sufficient details on this subject from different perspectives.
Most of the papers, including [24, 26, 27, 28, 30], although very informative,
only review the DL algorithms that are used or can be used for traffic
analysis and do not provide any sort of comparative analysis, which does not
help choosing the right method for different real-world traffic problems. The
relevant surveys we include are from 2013 and after; however, we want to
mention some papers, including [31, 32, 33] are outdated, but highly related
to our survey with the topics of CV-based traffic safety analysis. Table 2
provides more specific details on the main focus and the shortcoming of the
recently published survey papers.

In addition to covering newly published CV-based methods, our paper covers the
shortcomings of previous surveys and considers the video-based driving safety
problem from different perspectives. More specifically, we list exemplary
problems in video-based driving safety analysis; we review requirements and
challenges from an external observer’s perspective; we review datasets and
important industrial developments; we make connections to closely related
areas of utilizing crowd-sourcing, and edge and cloud computing for bulk
processing; we highlight connections to behavioral science, insurance
industry, and other policy maker entities.

The rest of this paper is organized as follows. Section 2 reviews mainstream
lines of DL methods used for vision-based driving safety analysis. In Section
3, data acquisition equipment and methods are reviewed. Section 4 includes
discussions about different stages of video pre-processing for safety analysis
by highlighting historical milestones, successful methods, current trends, and
remaining challenges for each category. Section 5 reviews DL methods for video
processing with application to traffic safety analysis. A short list of sample
problems in driving safety analysis is provided in Section 6. Section 7
reviews recent trends in deep learning that can influence the field if video-
based driving safety analysis. A list of commonly used datasets with
applications to traffic monitoring and traffic safety analysis is provided in
Section 8. Section 9 list a set of key safety metrics used for assessing the
potential for crash occurrence and crash severity.  Section 10 provides
different key points such as connection to other fields and potential
applications of safety analysis methods. A roadmap of this technology is
offered in Section 11. Remaining challenges and issues are discussed in
section 12.

##  2 Deep Learning Methods

The common core of vision-based driving safety analyses is using deep learning
methods for image/video processing. We skip the details of DL methods here for
the sake of brevity and refer the interested reader to previous reviews [34,
35].

It is noteworthy that most of the recent developments in DL have been driven
by two applications, Computer Vision (CV), and Natural Language Processing
(NLP), as two key representatives of visual and sequential processing
problems. Most elegantly designed DL platforms utilize Multilayer Perceptron
(MLP), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN),
and transformers as their building blocks. Table 3 provides a compact and
informative comparison of these methods. We will provide a more detailed
analysis of custom-built DL methods used in the context of driving video
analysis and safety control.

Table 3: A brief comparison among typical DL architectures.

Architecture | MLP | CNN | RNN/LSTM | Transformer
---|---|---|---|---
Pros | Straightforward to design |  | Appropriate for high-dimensional data
---
Learns features with locality
Shift-invariance
Fewer parameters
Powerful on CV tasks
| Appropriate for sequential data
---
Can learn long-term dependencies
| Processes sequences in parallel
---
Appropriate for long sequences
Captures long-term dependencies
Enables self-attention mechanism
Powerful on both CV and NLP tasks
Cons |  | Many parameters (dense connections)
---
Limited capability
| Do not encode the position of object
---
Gradient vanishing in deep architectures
(solved by ResNet and auxiliary output)
Weak on long sequential data
| Hard to train for gradient issues
---
Handles only serially fed data
Powerless on extreme long-term dependencies
| Many parameters (long training time)
---
Does not consider locality
Over complicated for short sequences
Inductive Bias | Weak | Locality/spatial invariance | Sequentiality/time invariance | Weak

Training a network is not always straightforward and can be impacted by many
factors, such as hyper-parameters (mini-batch size and learning rate). Due to
the time sensitivity of safety-related applications, fast training is a key
requirement. Table 4 lists some common tricks (e.g., dropout, regularization,
easy-derivative Rectified Linear Unit (ReLU) and leaky ReLU activation
function, data augmentation, and soft labeling) to accelerate the training
process and boost the recognition performance. For example, word embedding
uses low-dimensional space to present high-dimensional data allowing words
with similar meanings to be closer in the low-dimensional space. This often
outperforms the one-hot encoding. Such tricks can be translated from the NLP
context to driving behavior analysis. An exemplary scenario would be the
personalized driving anomaly detection. Suppose there are 20 collected
features for each driver. If there are 100 drivers, by one-hot encoding, the
input dimensions of data should be $100+20=120$ per time point, which leads to
the curse of dimensionality. In fact, difference among different drivers (or
driving styles) can be well represented by as few as 20 dimensions.

Table 4: Common tricks to accelerate DL training.

Name | Description
---|---
Data Normalization | Stabilize the training
Xavier/Kaiming Initi. | Initialize based on the magnitudes of signals
| Batch/Layer/Instance
---
Normalization
| Local response normalization
---
Regularize
Avoid gradient vanishing
Dropout |  | Reduce number of parameters
---
Avoid overfitting
L1/L2 regularization | Avoid overfitting
Weight decay |  | Similar to L2 regularization
---
but different in some optimizers (e.g. Adam)
LeakyRelu | Avoid dead neurons (as for Relu)
Word Embedding | Manage high-space categorical data
Soft Labeling |  | State the relations between different classes;
---
Improve generalization
Data Augmentation |  | Improve the variety of data
---
Improve generalization
Residual Connection |  | Avoid gradient vanishing
---
Allow building very deep networks
$1\times 1$ Conv | Combine channels
Gradient Clipping | Avoid gradient explosion

##  3 Data Acquisition

In this section, we investigate the role of data acquisition in developing
safety-related algorithms. First, we review data modalities and hardware used
for information acquisition. Next, we list the most commonly used datasets for
testing the developed algorithms.

Most traffic analysis platforms rely on data collected by different types of
sensors, including cameras, Global Positioning Systems (GPS), Radio Detection
And Ranging (Radar), and Light Detection and Ranging (LiDAR). These sensors
(except GPS) can be used in vehicles or on external observer systems such as
roadside infrastructures, drone-based aerial platforms, etc. The following is
a short description of sensors followed by a summary provided in Table 5.

Video Cameras are the most widely used means of information collection. Modern
cars are heavily equipped with cameras in various parts to capture imagery for
processing. Using thermographic cameras for night vision has become more
common than ever. Camera feeds can be used by the driver (e.g., backup camera)
to minimize safety risks, or by the control computer in an AV for automated
driving. The imagery can also be used by more advanced AI platforms for
driver’s cognition assessment in real-time mode (e.g., driver drowsiness
assessment [36], and distraction awareness [37]). Volume collections of
imagery can be stored for further analysis for transportation infrastructure
revisions.

GPS systems provide an accurate position of the vehicle by communicating to
GPS satellites. The accuracy of GPS depends on different factors, including
satellite geometry, signal blockage, atmospheric conditions, and receiver
design features/quality. The global average User Range Error (URE) of GPS can
be as low as 2.3 ft[38]. With recent advances in 5G wireless networking, even
preciser positioning is available for vehicles. For instance, Verizon launched
its Hyper Precise Location (HPL) using Real-Time Kinematics (RTK), with an
unprecedented accuracy of 1-2 centimeters[39].

Radar operates based on Doppler frequency shift in the reflected wave [40].
Radar units are often installed on modern cars, roadside poles, police
vehicles, and portable speedometer guns to measure the absolute or the
relative speed of other vehicles. Radars are low-cost and relatively robust
devices appropriate for different weather conditions and illumination
intensities. Some radar units may have a narrow Field of View (FoV) but are
capable of long-range detection [31]. A traditional radar is a single source-
detector, which does not have the spatial resolution required for precise
scene/environment description; however, in recent years, imaging radars have
been developed by adopting multiple-input, multiple-output (MIMO), and radar-
on-chip technologies. Although such advanced technology is still expensive,
attempts are made to reduce the cost of broader adoption[41].

LiDAR uses laser beam reflection to enable accurate positioning down to
centimeter’s scale [42]. The 3D scanning of multiple laser beams provides a 3D
point cloud image (3D map) of the surrounding obstacles with accuracy much
higher than regular radars [43]. LiDARs send out a near-infrared laser beam
and detect reflections from the object; thus, it can still operate in dark
conditions, in contrast to visual sensors. Its use is less common than radars
for a few reasons, including its higher cost, relatively sparse spatial
resolution, especially for a limited number of scanning laser beams, extremely
narrow FoV, and computational complexity of $360^{\circ}$ scanning, noting
that low-complexity point cloud methods are still under development. Solid-
state LiDARs have been developed to reduce the cost, while enhancing the
spatial resolution[44].

Drones are commonly used nowadays to implement aerial monitoring systems. Most
external observer systems utilize sensors in roadside infrastructures.
However, the use of drones, also known as Unmanned Aerial Vehicles (UAVs), is
gaining more attention in different applications to enable fast, low-cost, and
on-demand monitoring [45], and traffic analysis is not an exception.
Particularly, drones can provide top-view and clearer occlusion-free images of
the traffic flow when needed [46]. A network of cooperative drones can
collectively cover relatively large areas [47]. Modern drones may be equipped
with advanced sensing platforms, high-resolution cameras, and more importantly
advanced features such as learning-based image processing, AI-based autonomous
control, collision avoidance and auto lading, auto-calibration, real-time
transmission, object tracking, and image restoration. The key challenges of
drones are their limited payload, flight time, and communication range under
study by several research teams. Aerial images pose new challenges to the
research community, such as tackling image stabilization, small object
recognition, and developing lightweight ML algorithms customized for top-view
images.

Other Sensors: In addition to the aforementioned broadly-used sensors, there
exist some custom-built advanced sensors that can be used for traffic
monitoring and driving safety analysis. Passive Infrared sensors[48],
Inductive Loops[49], and Piezoelectric sensors[50] can enable measuring basic
flow parameters such as vehicle count, speed, and flow volume, noting that
Piezoelectric sensors are still used for weigh-in-motion measurement.
Environmental Sensor Stations (ESS)[51] on the roadway can collect atmospheric
data, including air temperature and humidity, visibility distance, wind speed,
wind direction, etc., as side parameters to be used for traffic analysis.

Table 5: Comparison of sensors. Here “$\surd$” means this kind of measure has
been implemented or available to complete.

Sensor Type |  Measures Type | Cost |  Shape Modeling |  Robust to Bad Environment444Noting that visual cameras may perform poorly under low illumination, and both visual cameras and LiDARs are limited by the bad weather, such as fog, dust, rain, or snow.
---|---|---|---|---
| Vehicle
---
Count
| Speed/Distance
---
Estimation
| Vehicle
---
Classification
| Pedestrian
---
Detection
Road feature
Camera (Visible) | $\surd$ | Hard | $\surd$ | $\surd$ | $\surd$ | Low | 2D | Low
Camera (Infrared) | $\surd$ | Hard | $\surd$ | $\surd$ | $\surd$ | Low | 2D | High
LiDAR | $\surd$ | Easy | $\surd$ | $\surd$ | $\surd$ | High | 3D | Middle
Radar | $\surd$ | Easy |  | $\surd$ |  | Moderate | weak | High
UAV Camera | $\surd$ | Moderate | $\surd$ | $\surd$ | $\surd$ | Low | 2D | Low

We recognize that the properties of sensors are varied. For example, visual
cameras require clean and high-visible environments and offer a richer set of
information in terms of color space and visual geometry, hence are more
appropriate for detection and classification applications. However,
interpreting imagery may require more computation powers. LiDARs, on the other
hand, provide more accurate precision for object detection and depth and speed
estimation but are expensive and less energy efficient. Some manufacturers
like Tesla prefer pure vision-based perception [52]. However, fusing multiple
sensors can be advantageous from the safety perspective and is adopted by more
vendors, as discussed in [53].

##  4 Video and Image Pre-processing

In this section, we review different stages of a typical video-based traffic
analysis framework and highlight key developments, historical milestones,
current trends, and existing challenges.

Super Resolution: The video and image super-resolution aims to reconstruct a
Higher Resolution (HR) result from a Low Resolution (LR) observation. Super-
resolution is a typical stage in image pre-processing and can be applied to
traffic imagery to enhance the performance of the subsequent learning tasks,
such as vehicle classification and license plate detection. One popular
supervised learning method is the DL-based Single Image Super-Resolution
(SISR) method which creates a mapping between the low and high-resolution
images by training a deep CNN. Most of the existing learning-based SISR
methods are trained and evaluated using simulated datasets [54, 55, 56, 57],
where the LR images are generated by applying a hand-crafted degradation
process into the HR samples. For instance, one may apply bi-cubic down-
sampling to the original HR samples to obtain LR results. Recently, more
advanced SISR [58] methods are developed for real-world applications with
unknown and more complicated degradation processes, which can be used as a
benchmark method for traffic image analysis as well.

In contrast to the simple spatial interpolation used in the SISR family for
image processing, Video Super-Resolution (VSR) methods utilize both spatial
and temporal relationships between consecutive frames to improve the quality
of the reconstructed videos[59]. These methods are essential in processing
roadside traffic videos, especially under poor visibility in foggy, rainy, and
cloudy weather conditions.

An important application of SR methods is license plate detection for vehicle
identification. Early works often focused on conventional signal processing
methods. For instance, [60, 61] deployed a Markov random fields-based method
for plate detection. [62] proposed a Gaussian Mixture Model (GMM) to enhance
the plate location and SR reconstruction. Compressed Sensing (CS)-based
methods[63, 64, 65] can also be used to address this task by enforcing the
sparsity of images in the frequency domain, which is equivalent to smoothness
in the spatial domain. Recently, DL-based SR algorithms are proposed, which
perform more accurately and efficiently. [66] is an example of such methods
which use a CNN architecture to convert a low-resolution license plate into a
high-resolution version. Some recent works [67, 68, 69, 70] tend to use
Generative Adversarial Networks (GAN) as their processing framework, which
achieves a higher performance using a more reasonable real-time loss along
with an adversarial loss, when inferring.

Denoising: This is another critical pre-processing task to compensate for
imaging artifacts and obtain clear and noise-free images before feeding them
into the subsequent learning modules. This is a critical step in processing
traffic imagery, especially when taken in motion or under low illumination and
poor environmental conditions like rainy, cloudy, and foggy weather.

Conventional methods typically use filtering, interpolation, and smoothing
methods either in time, frequency, or wavelet domains to remove noise from the
captured images. In contrast, newer methods use more advanced concepts such as
sparsity in the frequency domain, dictionary learning to model common noise
patterns, prior knowledge about the noise model, and noise pattern discovery
to more elegantly remove the noise from the captured images.

Traditional methods suffer from several shortcomings, including (1) involving
complex optimization methods in some cases, (2) the need for manual parameter
setting (e.g., the scale factor of Gaussian spatial filtering), (3) and using
a fixed model which deems inflexible in tackling different noise patterns and
ignore the learnability of some noises.

DL-based video and image denoising algorithms take advantage of the neural
networks to learn the spatial or temporal dependency between pixels to
reconstruct clean samples by end2end training and inferring. Therefore, DL
methods provide sufficient flexibility in adapting to different conditions. In
most research works on image denoising, a synthetic Additive white Gaussian
Noise (AWGN) model is adopted to simulate the noise and evaluate the
algorithm. Using a synthetic AWGN noise model has the clear advantage of
simplifying the testing phase and quantifying the noise impact. However, it
might oversimplify the problem since the real-world noise models can be more
complicated depending on the noise source. Further, one may benefit from
exploiting common noise patterns for more structured noises. For instance, the
noise caused by rainy conditions may need a different treatment than a noise
caused by the camera lens scratch. For instance, [71, 72] generates noisy and
clean image pairs by controlling the ISO (sensitivity to light) of the
cameras. By these approaches, the collected data could be used to emulate the
camera-related noise under real-world conditions.

Similar to SR methods, denoising methods are typically used for generating
clean traffic images that could be used to improve the precision of higher-
level tasks. For example, in [73], a low-rank decomposition image denoising
method is proposed for restoring the noisy traffic image. Likewise, in [74],
spatio-temporally denoised images are used to enhance the performance of the
traffic incident detection algorithm.

Table 6: Examples of fine-grained vehicle classification.

Method | Paper:[Ref] Authors (year) | Performance Accuracy [Dataset]
---|---|---
3D box+CNN |  [75]Sochor et al.(2016) | 83.20% on BoxCars116k(self)
3D box+Vgg-16 |  [76]Sochor et al.(2018) | 92.27% on BoxCars116k(self)
DenseNet-161 |  [77]Ma et al.(2020) |  | 93.81 % on Stanford Cars[78]
---
97.89 % on COMPCARS[79]
DenseNet-161 |  [80]Li et al.(2019) | 93.51% on Stanford Cars[78]
part-based +DenseNet-264 |  [81]Xiang et al.(2019) |  | 94.3% on Stanford Cars[78]
---
99.6% on COMPCARS[79]
Part-based+RCNN+SVM |  [82]Huang et al.(2016) | 87.3%
Faster R-CNN+Feature Fusion |  [83]Zhu et al.(2019) | Top 1: 79.1%, Top 5: 94.1% on COMPCARS[79]
CNN+Feature Fusion |  [84]Yu et al.(2018) | 98.89% on COMPCARS[79]
ResNet-50 + Attention |  [85]Yu et al.(2020) |  | 93.1% on Stanford Cars[78]
---
95.3% on COMPCARS[79]
Attention |  [86]Ke et al.(2020) |  | 94.5% mAP on Stanford Cars[78]
---
95.8% FZU dataset (self)
multi-task CNN |  [87]Hu et al.(2017) | Top 1: 91%, Top 5: 97.7% on COMPCARS[79]
CNN+filter bank |  [88]Wang et al.(2018) | 93.8% on Stanford Cars[78]
Fine-tuning Vgg-16 |  [89]Zhang et al.(2018) | 98.71% on COMPCARS[79]

Video Stabilization: Traffic videos may contain vibrations, especially when
captured by vehicle dashboard-cameras while driving on rough roads. This may
underline the performance of the subsequent processing stages (e.g., vehicle
detection, speed estimation, etc.). Digital video stabilization techniques are
proposed to improve the visual quality of the captured videos [90, 91]. The
common spirit of most video stabilization methods is extracting trajectory of
objects or their representative feature points between consecutive frames, and
the re-aligning the frames to smooth out the trajectories, based on the
assumption that noise-like high-frequency fluctuations, especially when shared
among most image descriptors, are caused by the camera shakes. We can
subdivide these methods into pixel-based and feature-based methods. The pixel-
based methods typically use block matching [92, 93], phase information [94,
95], and optical flow [96, 97] to estimate the camera motion. Feature point
detection methods can be used to convert high-dimensional images into low-
dimensional representations to reduce the computation overhead. In [98],
Scale-Invariant Feature Transform (SIFT) [99], Speeded Up Robust Features
(SURF) [100], and other feature point extraction methods are compared for
evaluating their impacts on video stabilization. Due to the computational
complexity of video stabilization methods, it is often performed offline,
which may not satisfy the requirements of real-time video-processing tasks.
Recently, DL-based methods [101, 102, 103] enable fast and accurate online
stabilization in almost a real-time fashion by instant processing of each
incoming video frame with low latency.

Most DL-based methods utilize CNN and similar architectures for video
stabilization using various sources of traffic data. In [104, 105, 106], video
stabilization was used for aligning the car-mounted camera captured videos. In
[107], the stabled UAV captured videos were used to obtain higher accuracy for
traffic video analysis.

##  5 Deep Learning for video processing

Deep learning methods are heavily used for video processing for their
outstanding power in solving different problems such as object detection,
object recognition, event recognition, and other video understanding tasks in
general. DL methods can be considered the brainpower of most AI platforms
developed for video-based traffic safety analysis.

###  5.1 DL-based Classification Methods in Traffic Analysis

Figure 4: The performance of some fine-grained models on Stanford Cars and
COMPCARS.

Classification is one of the most fundamental tasks in computer vision. The
use of classification in this context can be implemented for object
classification (e.g., classifying objects between vehicles, pedestrians,
motorcycles, traffic signs, etc.), traffic light and railroad crossing barrier
status check, and scene detection (e.g., buildings, roads, road lanes,
roadside infrastructures, etc.). Often, it is the backbone or the feature
extractor part of the detection networks (e.g., SSD[108]) and segmentation
networks (e.g., Mask R-CNN[109]). It can be developed at different levels,
such as the basic level for classifying different object types and fine-
grained classification into sub-categories (such as differentiating traffic
signs or identifying vehicle classes among sedans, SUVs, trucks, etc.) based
on the semantic content of the input. The well-known baseline classification
methods include AlexNet(2012) [110], VggNet(2014) [111], GoogleNet(2015)
[112], ResNet(2016) [113], MobileNets(2017) [114], DenseNet(2017)[115],
EfficientNet(2019), [116] etc. It is noteworthy that ResNet is the most highly
cited paper in all areas in Google Scholar Metrics 2020, which further proves
its extraordinary achievement.

Classification is often considered an upstream task. Therefore, frameworks of
the downstream tasks, such as detection, tracking, and segmentation, often use
the pre-trained versions of these baselines DL architectures as their backbone
to extract the hidden representations. Note that these baseline methods mainly
utilized CNN architectures until recently, when a framework named Transformer
has offered a new breed of deep learning methods with even higher
performances, as discussed in Section 7.1.

Since the use of classification methods for most traffic-related problems is
straightforward and noting the fact that there exist comprehensive reviews on
classification methods, we skip the review of classification methods and refer
the interested reader to [117, 118, 119, 120]. Here, we only review fine-
grained classification methods that are custom-built or customized for
traffic-related problems, as presented in Table 6 and Fig.4.

Figure 5:  (a),(b) are the examples of vehicle detection by two alternative
methods YOLOv5[121] and Faster R-CNN[122]. They output the location and
category of each object with detection confidence. Cars are shown by orange
bounding boxes, trucks are shown by lime bounding boxes, buses are shown by
green bounding boxes, and persons are shown by red bounding boxes. As shown in
(a),(b), YOLOv5 performs worse on small vehicle detection while Faster R-CNN
has missed objects in the near zone. (c) is the is PII removed data.

###  5.2 DL-based Object Detection Methods in Traffic Analysis

Object detection is another key stage in DL-based processing pipelines for
driving safety analysis. Object detection simply means locating different
objects in images and video frames, potentially with complex backgrounds, by
drawing bounding boxes around the objects of interest.It can coexist or be
integrated with object classification and labeling. An illustrative example of
vehicle detection using 2 benchmark methods is shown in Fig. 5(a)(b).

Notable examples of object detection in the context of driving safety analysis
include detecting surrounding vehicles, humans, traffic signs, and obstacles.
It also can be part of more complicated tasks such as traffic distribution and
composition analysis, improper lane crossing events, trajectory extraction,
speed estimation, moving object tracking, path planning, and detecting
vehicles on road shoulders, etc., as presented in Table 15.

Another use case of object detection is removing Personal Identifiable
Information (PII), such as masking human face and license plate numbers before
publishing traffic video, as shown in Fig. 5(c).

Although there exist some datasets for traffic analysis from the roadside
cameras [123, 124, 125], still there is a critical need for larger datasets
that cover different zones, urban, suburban, and rural setups, residential and
high-risk zones, railroads, and environmental conditions.

Some notable ongoing research problems include solving the trade-off between
the algorithm’s accuracy and speed, realizing small object detection,
distributed and federated learning, model sharing among roadside servers, and
implementing lightweight models and embedded devices appropriate for
autonomous vehicles, etc.

Figure 6: Network architecture for R-CNN family of localization and
segmentation.

Compared to the conventional object detection algorithms such as Viola-Jones
detector[126], the Histogram of Oriented Gradients (HOG detector)[127], and
Deformable Part-based Models (DPM)[128] ), CNN-based methods substantially
improve the recognition success rate.

From the implementation point of view, the DL-based algorithms can be divided
into one-stage and two-stage methods. The two-stage detectors first generate
Regions of Interests (RoIs) and then send the region proposals down the
pipeline for object classification and bounding-box regression. R-CNN
series[129, 130, 122, 131, 132] comprises the most popular two-stage algorithm
family. The architectures of R-CNN series are shown in Fig. 6. Specifically,
R-CNN [129] adopts Selective Search[133] to generate region proposals, and
then feed resized (by cropping or warping) proposals into a CNN-based backbone
to extract features. Finally, Support Vector Machine (SVM) performs
classification for objects’ categories and locations. SPPNet [134] introduced
Spatial Pyramid Pooling (SPP) layers, which solved the fixed-size input issue.
Integrated by R-CNN and SPPNet, Fast R-CNN [130] realized RoI pooling to
output fixed size features and adopted multi-task loss to allow single-stage
training. Since Fast R-CNN was still bottlenecked by the Selective Search
heavy computation, Faster R-CNN [122] introduced Region Proposal Network
(RPN), an end-to-end trainable network to generate quality region proposals.
Mask R-CNN adds a parallel branch to predict the object mask.

One-stage detectors directly treat object detection tasks as a regression and
classification problem. These methods are divided into anchor-free and anchor-
based methods. In anchor-based methods, a set of bounding boxes with different
predefined sizes are required to capture the scale and aspect ratio of the
objects. Some famous implementations include You Only Look Once (YOLO) family
(e.g., YOLOv2 [135], YOLOv3[136], and YOLOv4[137], and YOLOv5 [121]555Note
that the authors of YOLOv5 are different from the previous versions, as well
as the Single Shot multi-box Detector (SSD) series[108]). The basic idea of
YOLO is demonstrated in Fig. 7. The network treats the detection as a
regression problem and learns the location and class of each bounding box
separately. Specifically, an image is split into $S\times S$ patches. In each
patch, the network can predict the coordination of $B$ bounding boxes with
their confidence levels as well as the class of each patch. After aggregation,
the bounding box of an object is localized with its category. Then, non-
maximal suppression is applied to filter out the extra bounding boxes for the
same object.

Figure 7: The demonstration of the basic idea of YOLO.

Recently, anchor-free methods are getting more attention to avoid defining
anchor-related hyperparameters and to ease complicated computations. The main
ideas include implementation by dense prediction (e.g., DenseBox[138], Fully
Convolutional One-Stage (FCOS) object detectors [139], RetinaNet[140]) as well
as implementation by keypoints and center points (e.g., CornerNet[141, 142],
CenterNet[143, 144],ExtremeNet[145]).

Generally speaking, two-stage methods can achieve higher accuracy but at lower
speeds than the one-stage methods. Some recent one-stage methods (including
YOLO v4[137] and SSD[108]) solve the trade-off between the accuracy and speed
by realizing a more efficient network structure. A summary of these
algorithms’ performance is presented in Fig.8.

Figure 8: Some object detection models perform on MS COCO test-dev.

The applications of object detection methods in traffic video analysis mostly
relate to understanding the objects surrounding the road users, such as
vehicles, plates, and traffic signs. Most research works in this area adopt
one of the aforementioned algorithms for object detection, as summarized in
Table 7. The metrics used for object detection, recognition, and image
segmentation are outlined in Table 9. We note that some works (such as [146,
147, 148, 149, 150]) tend to fine-tune the CNN framework according to the task
requirement, and the recent mainstream traffic works begin to deploy the R-CNN
series, YOLO series, and SSD widely. It means these algorithms can stand the
test of practice, but it does not mean that the other algorithms are not
favorable. It also can be due to the relatively low complexity of this task,
or due to the difficulty of deploying recent detectors in real-world
scenarios.

Pedestrians are often the most vulnerable entities on the road, hence
pedestrian detection is usually considered a top-priority component of safety
assessment and control systems. Pedestrian detection, especially in crowded
areas, would be more complex than similar object detection tasks. Indeed,
early CNN-based frameworks underperformed on the task due to the uncertain
dense distribution and high dynamics of pedestrians. Some common issues,
including the multi-scale problem (different receptive fields with low-
information and noisier observations for small-scale pedestrians) and
occlusion by crowding (parts of pedestrians are invisible), may challenge
these models and decline their accuracy [151]. Some solutions, such as [152,
153] addresses the multi-scaling issue by modifying the scale of the region
proposals. Specifically, MS-CNN[152] has multiple layers to output features
with different receptive fields at the region-proposal sub-network. SAF R-CNN
[153] uses multiple sub-networks to learn the large-scale features and small-
scale features, respectively. Likewise, [154] uses an attention mechanism by
applying a fine-grained attention mask to focus on differently scaled
pedestrians. SSA-CA [155] performs detection after semantic segmentation.

Straightforward methods (such as [156, 157]) are proposed to solve the
occlusion by enhancing the model to learn patterns with different occlusions.
High-level semantic features are employed in [158, 159] to help the occlusion
detection. The other explicit solutions [160] use data augmentation by
introducing extra annotation of the visible parts in the training phase, which
substantially enhances the detection ability even under high occlusions. From
the author’s perspective, the work in [161] is very creative, which uses a
Cycle-consistent Adversarial Network (Cycle-GAN) [162] to translate front view
images into bird view images, then solve the detection problem by decomposing
it into three subtasks of pedestrian localization, scale prediction, and
classification. It achieves an outstanding performance of 6.37 Miss Rate (MR)
on Caltech (reasonable) and 9.3 MR on CityPersons (reasonable) datasets.

We listed some recent works for pedestrian detection in Table 8. These methods
for pedestrian detection are often evaluated on Caltech [163] and CityPersons
[163] datasets, which both include person-to-person occlusion caused by
crowding and other objects. It is notable that these two datasets are designed
specifically for pedestrian detection. A more complicated scenario of
detecting vehicles and pedestrians simultaneously, is often validated using
auto-driving datasets, such as KITTI [164] (see Table 18).

Figure 9: A dual feel camera feed (visible and thermal image pair) for multi-
spectral pedestrian detection. Images are from [165].

It is noteworthy that some handcrafted features, such as optical flow
features, are complementary to deep convolutional features, which can further
boost the models’ ability [166].

Another interesting problem is pedestrian detection from multi-spectral images
(such as RGB true-color + thermal camera feeds, as shown in Fig. 9. This
approach can be advantageous because infrared images provide more informative
and robust features under low illuminations and extreme weather conditions.
These works often require feature fusion to boost the overall performance.
Feature fusion can be implemented in different stages, ranging from low-level
(near the input layer) to high-level (near the output layer). The authors of
[167] have shown that fusion at the middle-level (the authors name it halfway
fusion) often achieves the best results. More recently, attention mechanisms,
including channel attention and semantic attention, are applied for feature
fusion [168] to allow the network automatically learn the importance of
different features. Alternatively, an illumination-aware weighting mechanism
is used in [169] to learn the variation of illumination and adaptively mix the
features of the visible (RGB) and thermal images. Some minor issues such as
modality imbalance and weak alignment between different channels should be
taken care of when fusing multiple camera feeds [170, 171]. A summary of
recent implementations is presented in Table 8. KAIST [165] and CVC-14 [172]
datasets are often used as the benchmark validation dataset for these works.
More datasets for pedestrian detection are provided in Table. 18.

Table 7: Examples of object detection tasks for traffic analysis. If the
dataset is not indicated, it means the current work uses the dataset generated
by its authors. $\triangle$’ means that this work uses fine-grained detection
problem. ’*’ Means this work is based on YOLOv5. It is worth mentioning that
YOLOv5 may not be considered as a member of YOLO family.

Task | Methods | Paper: [Ref] Authors (year) | Performance: Accuracy [Dataset]
---|---|---|---
Vehicle Detection | R-CNN family |  |  [173]Espinosa, et al.(2017)
---
[174]Wang, et al.(2017)
[175]Soin, et al.(2017)
[176]Zhang, et al.(2017)
[177]Peppa, et al.(2018)
[178]Yu et al. (2017)$\triangle$
| 70%
---
84.43% on DETRAC dataset [179]
100%
96.5%
97%
98%
| YOLO family |  |  [180]Sang, et al.(2018)$\triangle$
---
[181]Kim, et al.(2019)
[182] Kasper-Eulaers et al.(2021)*
[183] Nayak et al.(2019)$\triangle$
| 99.51%
---
85.29%on UA-DETRAC[179]
93% car, 63% truck front, 52% truck back (winter condition)
99.73%
| SSD |  |  [184]Zhang, et al.(2019)
---
[185]Chen, et al.(2020)
[186]Cao, et al.(2020)
[177]Peppa, et al.(2018)
| 77.94% mAP on UA-DETRAC[179]
---
84.5%on KITTI
92.18% on KITTI
98.2%
|  | CNN
---
CNN
|  [146]Chen, et al.(2014)
---
[187]Zhou et al.(2016)$\triangle$
| 99.7%
---
62.85% on UTS(self), 64.44% PASCAL VOC2007[188], and 79.41% on LISA 2010[189]
Plate Detection | R-CNN family |  [190]Lee, et al.(2016) | 99.94%
| YOLO family |  |  [191]Kessentini, et al.(2019)
---
[192]Khazaee, et al.(2020)
[193]Xie, et al.(2018)
[194]Chen, et al.(2019)
| 97.67% on GAP-LP dataset[195] 91.46% on Radar
---
97.9%
98.32% on UCSD 97.38% on PKU[196]
98.22% on AOLP
| SSD |  |  [197]Rene, et al.(2020)
---
[198]Hu, et al.(2020)
[199]Danilenko, et al.(2020)
| 92%
---
90.12%
94+%
|  | CNN
---
Pre-pocessing+CNN
CNN
|  [148]Kim, et al.(2017)
---
[149]Selmi, et al.(2017)
[150]Masood, et al.(2017)
| 98.39% on Caltech[200]
---
94.8% on Caltech[200]
99.09% on US[201] 99.64% on EU[201]
| Traffic Sign
---
Detection
R-CNN family |  |  [202]Qian, et al.(2016)
---
[203]Shao, et al.(2019)
[204]Zhang, et al.(2020)
[205]Zuo, et al.(2017)
[206]Wu, et al.(2019)
[207]Peng, et al.(2016)
| 85.58%
---
69.56% on GTSDB+CTSD[208]
98.7% on GTSDB
34.49% mAP on CCF2016
mAP 91.75% GTSDB
mAP 90% on GTSDB
| YOLO family |  |  [209]Zhang, et al.(2017)
---
[210]Tai, et al.(2020)
[211]Liu et al.(2021)*
[212] Qin et al.(2021)*
| 96.69% on CTSD and GTSDB
---
99.1% mAP
97.2% mAP
94.3 % mAP
| SSD |  |  [213]Gao, et al.(2019)
---
[214]You, et al.(2020)
| 91.0% on GTSDB 75% on TT100K
---
|  | MSER+SVM+CNN
---
FullyConv
CNN
CNN
|  [208]Yang, et al.(2015)
---
[215]Zhu, et al.(2016)
[216]Wu, et al.(2013)
[217]Shustanov, et al.(2017)
| 98.24% GTSDB 98.77% CTSD
---
91% TT100K
AUC 99.73% “danger” , 97.62% "mandatory" on GTSDB
99.94% on GTSDB

Table 8: Examples of pedestrian detection methods with their miss rate.

Task | Method | Paper: [Ref] Authors (year) | Performance: Accuracy [Dataset
---|---|---|---
Single-spectral | Faster R-CNN |  [152] Cai et al. (2016) | 10% MR on Caltech
Fast R-CNN |  [153] Li et al. (2017) | 9.32% MR on Caltech
CNN+Attention |  [154] Lin et al. (2018) | 7.84% MR on Caltech
Faster R-CNN |  [218] Zhang et al. (2018) | 4.1% MR on Caltech;11% MR on CityPersons;
CNN+LSTM |  [158] Wang et al. (2018) | 8.4% MR on Caltech
FCN |  [159] (2019) | 3.8% MR on Caltech; 11.4% MR on CityPersons
Faster R-CNN+ Attention |  [155] (2019) | 6.27% MR on Caltech;
Cycle-GAN |  [161] (2020) | 6.37%MR on Caltech; 9.3% MR on CityPersons
CNN+GCN |  [157] Xie et al. (2021) | 6.4% MR on Caltech; 9.3% MR on CityPerson
FCN+pattern-parameter matching |  [219] et al. (2021) | 3.3% MR on Caltech; 10.4% MR on CityPersons
Multi-spectral | Faster R-CNN+ Fusion |  [167] Liu et al. (2016) | 25.73% MR on KAIST
Faster R-CNN+ Fusion |  [169] Li et al. (2019) | 15.73% MR on KAIST
CNN+ ATT+ Fusion |  IATDNN[34] [220] Guan et al. (2019) | 14.93% MR on KAIST
CNN+Attention+ Fusion |  [168] Zhang et al. (2019) | 14.12% MR on KAIST
Faster R-CNN+ Fusion |  AR-CNN[35][171] Zhang et al. (2019) | 9.34% MR on KAIST
SSD+ATT+ Fusion |  MBNet 20[170] Zhou et al. (2020) | 8.13% MR on KAIST
CNN+memeory learning |  [221] Kim et al. (2021) | 8.26% MR on KAIST; 23% MR on CVC-14
CNN+ATT+Fusion |  [222] Kim et al. (2021) | 7.89% MR on KAIST; 18.7% MR on CVC-14
YOLOv5+ATT |  [223] Jiang et al. (2022) | 7.85% MR on KAIST

Table 9: Evaluation metrics for different vision-based traffic analysis tasks
(detection, classification, etc.)

Metric | Definition
---|---
Accuracy | $\frac{\text{Correct predictions}}{\text{Total predictions}}$
Recall (R) |  $\frac{TP}{TP+FN}$ (*)
Precision (P) | $\frac{TP}{TP+FP}$
F1 score | $2\times\frac{P\times R}{P+R}$
Pixel Accuracy (PA) | $\frac{TP+TN}{TP+TN+FP+FN}$
Miss Rate (MR) | $\frac{FN}{TP+FN}$
Normalized error (mean) |  | Normalized error=$\frac{|d-d_{gt}|}{d_{gt}}$.
---
$d$ is the calculated distance and
$d_{gt}$ is the ground truth of distance.
Average Precision (AP) |  $AP=\int_{0}^{1}P(R)dR$ P-R CURVE (for one class)
mean Average Precision (mAP) | $mean(AP_{i})$
Intersection over Union (IoU) |  | $\frac{B_{p}\cap B_{gt}}{B_{p}\cup B_{gt}}$
---
$B_{gt}$: ground truth bounding box
$B_{p}$: predicted bounding box
Disparity accuracy[224] |  $disparity_{inlier}\%$. The disparity $\leq$ 3px/5% is true
Stixel-wise percentage accuracy | Similar to PA
Identification F1-Score ($ID:F1$) | $ID:F1=\frac{2ID:TP}{2ID:TP+ID:FP+ID:FN}$
multiple object tracking accuracy (MOTA) |  | $MOTA=1-\frac{FN+FP+\Phi}{T}$
---
$\Phi$ denotes the number of fragmentations
Multi-camera Tracking Accuracy (MCTA) |  | $MCTA=\underbrace{\frac{2PR}{P+R}}_{F_{1}}\underbrace{(1-\frac{M^{w}}{T^{w}})}_{\text{within camera}}\underbrace{(1-\frac{M^{h}}{T^{h}})}_{\text{handover}}$
---
$M^{w}$: within-camera identity mismatches
$T^{w}$: true within-camera detections
$M^{h}$: handover mismatches
$T^{h}$: true handover detections
RMSE | $RMSE=\sqrt{\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2}/N}$

###  5.3 Visual Tracking

Visual tracking refers to capturing the movement of specific objects by
processing video frames. Traffic video processing by autonomous vehicles or
traffic monitoring systems is perhaps the most popular application of visual
tracking. With the advent of AVs and technology-assisted driving systems, this
research area has become a hot topic. The mainstream algorithms can roughly be
categorized into Correlation Filter-based Trackers (CFT) and non-CFT methods
[225]. The two challenging issues are the natural difficulty of reliable
visual tracking and the lack of well-annotated datasets, especially for
driving safety analysis.

Visual tracking methods can be categorized into discriminative methods [226,
227, 228, 229] and generative methods [230, 231, 232, 233, 234] from the
modeling standpoint. Generative tracking methods comprise the following
sequential steps: (i) extract the target features to learn the appearance
model, which represents the target, (ii) search through the image area to find
areas that best match the model, using pattern matching. The target
information carried by generative models is often richer than that of the
discriminative methods. Also, generative models make it is easier to meet the
evaluation criteria and the real-time requirements of target tracking when
processing a massive amount of data. The key components of generative methods
include target representation and target modeling. The limitations of this
approach include (i) background information of the image is not fully
utilized, and (ii) the appearance of the target in different video frames may
include substantial randomness and diversity, which affects the stability of
the model.

Discriminative methods turn the tracking problem into a classification
problem, and rely on training a classifier to distinguish between the target
and the background. The target area is considered the positive sample in the
current frame, and the background area is the negative sample. Discriminative
methods turn the tracking problem into a classification problem, which can
simultaneously utilize the information from the target and background. ML
methods can be employed to train a binary classifier to distinguish between
the target (considered as the positive sample) and the background (considered
as the negative samples), and can be updated online as more video frames are
accumulated. The trained classifier is used to find the optimal area in the
next frame. Discriminative methods are often more robust than generative
models when facing appearance and environmental changes.

Correlation Filter (CF) methods and DL methods are often considered
discriminant methods. In recent years, tracking methods based on correlation
filtering [235, 236, 237, 238, 239, 240] have gained a lot of attention from
computer vision researchers because of their fast speed and reasonable
performance.

The correlation filter is initialized by the given target in the first frame
of the input. A classifier is trained by regressing the input features to the
target Gaussian distribution. The response peak in the predicted distribution
is found in the follow-up tracking step to locate the target’s position.
Correlation filtering methods, when combined with deep features and CNN
architectures (e.g., R-CNN series [241, 242, 243], SSD-based methods [244])
exhibit outstanding performances and hence have gradually become a dominant
approach in this field.

In addition to CF-based trackers, some other DL frameworks, even without using
correlation filters, can also achieve an excellent performance. More elegant
methods, including [245, 246, 247, 248, 249, 250] tried to directly employ
sequential learning models such as Long Short Term Memory (LSTM) or Recurrent
Neural Networks (RNN) in their network structures after CNN-based feature
extractors to capture the temporal information of video frames that represent
object motions.

Figure 10: The architecture of basic Siamese neural network for similarity
training.

Siamese Network structure (shown in Fig. 10) offers a paradigm shift for
visual tracking. The core idea of Siamese networks [251, 252, 253, 254, 255,
256, 257] is training twin networks to identify the similarity between two
different images, such as the same object in consecutive video frames. These
methods address both the similarity knowledge learning and the real-time
operation requirements with acceptable tracking accuracy.

Many contemporary traffic monitoring systems still tend to use conventional
tracking methods such as filtering and convolution methods due to their
efficiency, low complexity, and stability. For examples, methods based on
Kalman filtering [258, 259, 260, 261, 262], Gaussian Mixture Models (GMM)
[263, 258], Hidden Markov Models (HMM) [264], and SIFT-based methods[265, 266]
are used in traffic monitoring systems. However, using DL-based methods is
gaining traction in recent years to perform traffic monitoring tasks due to
the emergence of powerful and low-cost processing platforms, making DL methods
more affordable and near real-time.

Hybrid methods that enable tracking by detection are another possibility. For
instance, a DL method can be used for fast and accurate object detection,
followed by a second estimator based on conventional methods, Kalman
filtering, and Kanade–Lucas–Tomasi (KLT) feature tracker for tracking
purposes. This approach simplifies the processing job for object detectors
designed for specific tasks, such as multi-spectral pedestrian detection.

A summary of some important implementations is presented in Table 10. Note
that datasets KITTI, and MOT 15 Challenges [267] include pedestrians, in
addition to vehicles. Several studies, including [268, 269, 270], are
specifically designed to improve the safety of humans.

Table 10: Some traffic-related visual tracking method that use deep learning.
The default dataset is generated by the author of each work. The default
performance metric is accuracy unless specified otherwise.

Paper: [Ref] Authors (year) | Methods | Performance
---|---|---
[271]Qiu, et al.(2018) | YOLO+KLT | 76.4% Recall 88.2% Precision
[272]López-Sastre, et al.(2019) | Faster R-CNN |  | 30.5AP on subset:M-30 66.2%AP on subset:M-30-HD
---
38.1%AP on subset:Urban1 of GRAM-RTM
[268]Scheidegger, et al.(2018) | CNN+PMBM filter[273] | 80.04% MOTA on KITTI
[269]Zou, et al.(2019) | Siamese network+SPP+MDP |  | 75.29% MOTA campus
---
76.06% MOTAurban
78.14% MOTA highway
on KITTI
[274]Li, et al.(2019) | FPN+tracking loss | 83.2% IDF1 on nivida AI city
[275]Nikodem, et al.(2020) | hourglass |  | MOTA 97+%
---
MCTA 91+%
[276]zhao, et al.(2018) | SSD+dual Kalman filters | Shown in the form of figures
[277]Wang, et al.(2019) | YOLOv3+SIFT | 81.3% MOTA
[278]Usmankhujaev, et al.(2020) | yoloV3+kalman filter |  | 100% Daytime Wrong cases detection
---
89.83% Daytime(flipped) Wrong cases detection
86.11% Nighttime(flipped) Wrong cases detection
[279]Kwan, et al.(2018) | ResNet | For compressive measurements
[270]Fernández-Sanjurjo,et al.(2019) | CNN detector+DCF+Kalman filter | 86.96% MOTA on MOT 15[267]

###  5.4 Semantic Segmentation

Figure 11: A example of different types of segmentation. Source from [280].

Semantic segmentation, where objects of different types are separated, can be
considered the heart of many video processing tasks. For instance, vehicle
detection, vehicle tracking, and environment perception in a crowded
environment with interlaced and overlapping objects can be powered by semantic
segmentation when regular segmentation methods fail to separate objects from
complex backgrounds. An example of semantic segmentation is shown in Fig.
11(b). The purpose of semantic segmentation is to label each pixel of an image
to represent different categories (e.g., cars, pedestrians, roadside
infrastructures, traffic signs, etc.). For instance, semantic segmentation can
be employed by an autonomous vehicle for background modeling, identifying road
boundaries and free spaces, and detecting lane markings and traffic signs.
Semantic segmentation can also be used by an external traffic monitoring
system for analyzing the behaviors of human-driven and self-driving vehicles
in specific zones and times. To avoid reliance on massive data collection and
expensive annotations, semi-supervised and weakly-supervised learning methods
[281, 282, 283, 284] are developed for low-cost implementation with reasonable
performance.

Early works tended to deploy existing classification algorithms at the patch
level for semantic segmentation. Since 2014, Fully Convolutional Network
(FCN)[285] was introduced that allows spatially dense prediction tasks by
translating famous DL architectures such as AlexNet, VGG net, and GoogLeNet
into fully convolutional network architectures. Afterward, many upgraded
architectures such as U-net[286] and SegNet[287] were proposed, which build
upon the concept of FCN and utilize auto-encoder architectures for semantic
segmentation with small training datasets using data augmentation methods.
This architecture is further updated to multi-stage auto-encoder networks in
[288]. In some work[289, 290, 291, 292], atrous/dilated convolution
architectures are used to keep spatial resolution and expand the receptive
fields. The authors of [293] expanded the receptive fields by employing large
kernel with its proposed global convolution.

Figure 12: Semantic segmentation models perform on Cityscapes.

In [294, 295, 296], a feature fusion method is deployed that allows the
framework to learn global features merged with more local features. For these
methods, Conditional Random Fields can be used to enhance the output. Other
works [297, 298] are based on a Recurrent Neural Network (RNN) structure that
can better tackle sequence-related tasks. Additional models including[299]
based on graph convolutional network, [300] based on pyramid pooling model,
and [301] based on learning relations between the object region and pixels
have competitive performance. Furthermore, [302, 303, 304] focused on semantic
segmentation with 3D point cloud data, which have great potentials for
autonomous driving and traffic safety analysis. More specifically, the methods
[302, 303] are evaluated by indoor scene dataset, while [304] is evaluated by
urban scene captured by scanners, something potentially more relevant for
traffic analysis. The performance of some models is shown in Fig. 12.

The applications of semantic segmentation frequently appear in the world of
AVs. Some examples include (i) scene understanding, which involves
understanding the traffic environment with road users, (ii) free space
estimation to determine the available spaces on the road that a vehicle is
allowed to use with no collisions, and (iii) Stixel representation, which
assigns each pixel with a 3D depth information. A summary of related works is
presented in Table 11.

Table 11: Some semantic segmentation works in the traffic field.

Task | Paper: [Ref] Authors (year) | Methods | Performance
---|---|---|---
Scene understanding |  |  [305]Romera, et al(2017)
---
[306]Lyu et al.(2019)
[307]Deng, et al.(2017)
[308]Sáez, et al.(2018)
[309]Kendall, et al.
| Autoencoder
---
Edge Detection Network+Fusion
CNN+pyramid pooling module
Autoencoder
Bayesian Autoencoder
| pixel accuracy$>$95% Cityscapes
---
63.2% mIoU Cityscapes
54.5% mIoU Cityscapes
59.3% mIoU Cityscapes
63.1% mIoU CamVid
Free space estimation |  |  [310]Ohgushi, et al.(2020)
---
[311]Hua, et al.(2019)
[312]Levi, et al.(2015)
[313]Deepika, et al.(2017)
| Autoencoder
---
Autoencoder+fusion+optical flow
CNN
SegNet
| 21.9% mIoU
---
Path planning accuracy
0.15m Indoor and 0.27m Outdoor
0.87 AUC
89.12% maxF SEGMENTATION on KITTI
0.9667 IoU
Stixel representation |  |  [314]Schneider, et al.(2016)
---
[315]Cordts, et al. (2017)
| FCN+SGM
---
FCN+SGM+graphical model
| 7.8 Disparity Error on KITTI 15.2% on Ladicky
---
83.1% exact Disparity accuracy

###  5.5 Instance Segmentation

Instance segmentation, which deals with detecting and delineating distinct
objects of interest in images and video frames, can be considered as one of
the most difficult tasks in computer vision (Fig. 11(c)). It goes one step
beyond semantic segmentation and not only labels the pixels based on their
object categories, but also distinguishes between different object instances
of the same type. This is of crucial importance for traffic imagery analysis
in dense zones, where different objects (e.g., vehicles, pedestrians) have to
be identified and located in video frames for optimal decision making by AVs,
or to extract safety metrics by monitoring systems. In contrast to semantic
segmentation, instance segmentation only needs to find the edge contour of the
object of interest with no need for bounding boxes, hence it can realize a
more accurate object detection when assessing the behaviors of vehicles.
Building a reliable and real-time method, for instance, segmentation,
especially for crowded zones and under highly distorted traffic videos (e.g.,
in rainy and cloudy weather conditions), can be challenging.

Instance segmentation can be divided into one-stage and two-stage methods.
Two-stage methods often require generating region or object proposals followed
by a classification-based segmentation performed over the features extracted
from the selected regions or bounding boxes around object proposals. To
generate region proposal, [316, 317, 109] predict a bounding box for each
instance, while [318, 295, 319, 320] develop pixel-wise coarse segmentation
masks.

Since instance segmentation partitions the image by masking the detected
objects and associating each pixel with a distinct object, some two-stage
detectors such as Faster R-CNN [122]) can execute instance segmentation task
after some post-processing, e.g., by adding a branch for mask predictions.

One-stage instance segmentation methods do not utilize a separate stage for
generating region proposals; rather, they apply the segmentation directly to
the original images. Some methods [321, 322, 323, 324] inspired by one-stage
detectors (such as YOLO[135]) directly predict bounding boxes. However, these
anchor-based methods heavily rely on predefined anchors, which may be affected
by many factors such as the predefined anchor boxes’ aspect ratio and scales.
Another approach is developing anchor-free methods using dense prediction or
centerpoint/keypoints. For instance, [325, 326, 327] relied on FCOS[139] as
their dense prediction detector, while ExtremeNet[145] is a keypoint-based
detector that can roughly perform the segmentation task. The difference
between the accuracy of one-stage and two-stage methods is not as great as one
may expect. Indeed, recent one-stage methods such as CenterMask[326] can
perform real-time inference with accuracy as high as two-stage methods, or
even better. The performance of some instance segmentation models is shown in
Fig.13.

Figure 13: Instance segmentation models perform on MS COCO test-dev

In the context of traffic analysis, instance segmentation can be used not only
to identify and locate vehicles but also to obtain detailed information about
the vehicle, such as a vehicle’s class, number of axles, 3D bounding box,
dimension, etc. It can be used as part of the processing to identify road
lanes, traffic volume, etc. For instance, one may consider each lane as an
instance. Some famous methods that utilize instance segmentation for traffic
image analysis are presented in Table 12.

Table 12: Some instance segmentation methods used for traffic imagery
analysis. If the dataset is not indicated, it means that a proprietary dataset
generated by the authors is used.

Task | [REF] authors (Year) | Methods | Performance
---|---|---|---
| Obtain comprehensive
---
vehicle information
|  [328]Zhang, et al.(2015)
---
[329]Mou, et al.(2018)
[330]Zhang, et al.(2020)
[331]Huang, et al.(2018)
| CNN+MRF
---
ResFCN
Mask R-CNN
Mask R-CNN
| 59.0% object recall,
---
83.1% of the randomly sampled foreground pixel pairs ordered on KITTI
95.87% F1 score on ISPRS
97% vehicle types accuracy
1.333 front 4.698 side-way KITTI
Lane detection |  |  [332]Neven, et al.(2018)
---
[333]Roberts, et al.(2018)
| multi-task CNN
---
SegNet
| 96.4% accruacy
---
IoU Mapillary 82.9% CityScapes 85.2%
KITTI 83.8% author’s dataset 95%
3D reconstruction |  [334]Hadi, et al.(2020) | Mask R-CNN |  | AP@50% 8.862% CityScapes 43.949%
---
IDD[335] 5.643% WildDash[336]

###  5.6 Video-based Event Recognition

Video-based event recognition extends the role of DL methods from object
detection and identification paradigms into a more intricate problem of
understanding events. It provides endless possibilities to explore the
interactions among objects in an interactive environment rather than focusing
on disjoint object-based tasks. Indeed, without event detection and analysis,
the majority of video information remains unexploited and underutilized. Note
that many safety metrics relate to the interactions of vehicles with one
another and with the environment. For instance, traffic sign interpretation
can be recast as an image-based object classification problem, while more
intricate violations of traffic safety such as unsafe lane changing behavior
without signaling require modeling interactions between vehicles and their
surrounding environments. Such challenging problems are still in their infancy
stages and apparently require heavy investment by the research community. In
essence, the event recognition problem is also related to another well-
investigated problem of video-based human action recognition, and similar
tools and algorithms can be adopted here.

An alternative method of event analysis is using conventional analysis
approaches by extracting object-based information and manually feeding them
into statistical and reasoning models such as Hidden Markov Models (HMM) for
safety analysis. However, with the recent advances in developing powerful DL
methods, they enable a more automated and direct way of evaluating the
behaviors of involved vehicles. The most naive way of event recognition can be
realized by extracting static features from video frames using methods like
SIFT [337], SURF [100], the Local Binary Patterns (LBP) [338, 339], HOG
detector [127], Binary Robust Invariant Scalable Keypoints (BRISK) [340],
Features from Accelerated Segment Test (FAST) [341, 342] and GIST (a very low-
dimensional scene representation) [343, 344], and then performing object
detection followed by a time-series analysis for even recognition. Some other
methods combine the feature extraction and time-series analysis stage into
directly extracting temporal features using methods like motion spatio-
temporal features (Motion SIFT (MoSIFT) [345], Spatio-Temporal Interest Points
(STIP) [346], and Dense trajectories [347, 348]), then perform the
classification task.

More contemporary event detection algorithms benefit from DL methods to
automate this process, and a dominant method is directly applying a 3D
convolutional network to process non-anomalous frames [349]. Another approach
is deploying CNN to extract spatial features and then performing sequential
analysis using methods such as RNN/LSTM [350, 351, 352, 353, 354, 355, 356] to
preserve the temporal features.

Most of these methods use supervised learning methods to detect a set of
predefined events. Therefore, developing generic methods for understanding
safety risks from driving profiles and tackling unseen types of safety
violations has a long way ahead.

Event recognition is of particular importance for traffic safety analysis
since it can be used for detecting abnormal events and traffic violations and
their associations with crash rates [357], car behavior analysis [358, 359],
and pedestrians’ crossing identification [360, 354, 361]. Recently, some
works[362, 363, 364] try to predict anomaly actions using Generative
Adversarial Networks (GAN). The core idea is predicting the future video
frames for a normal user with rational behavior from the history of normal
sequences to identify severe abnormalities by comparing the observed video
frame against the anticipated one.

For vehicle-level analysis, when the goal is detecting plain and simple
events, conventional methods achieve a reasonable performance. For instance,
[365] uses HMM to detect traffic abnormality, and [366] deployed a topic model
to recognize crashes from surveillance videos. However, using DL methods can
be used to analyze more complex events and achieve higher performance records.
Some generic platforms like Rekall [367], provide a query-based approach to
translate the challenging task of event recognition into a sequence of object
detection and classification problems.

We summarize important DL-based event recognition methods in Table 13, and
present some popular datasets for event recognition tasks in Table 18. One
observation is that unsupervised, semi-supervised, and self-supervised models
are becoming more prevalent in recent works [357, 368, 364] to mitigate the
costly and tedious job of video annotation and simplifies volume video
processing.

Table 13: Some recent DL works for video-based traffic abnormal event
detection. Unless otherwise specified, the non-indicated numbers is the
accuracy.

Paper: [Ref] Authors (year) | Methods | Performance
---|---|---
[364]Nguyen, et al.(2020) | GAN | F1 score 0.9412 AI City Challenge 2019
[368]Yao, et al.(2019) |  | Auto Encoder-Decoder
---
with GRU
60.1% AUC on A3D Dataset
[369]Tian, et al.(2019) | YOLO-CA | 90.02% AP CAD-CVIS
[370]Kim, et al.(2020) | 3D conv | 82%
[371]Ijjina, et al.(2019) | 3D conv | 71%
[372]Shah, et al.(2018) |  | DSA-RNN[373]
---
Faster R-CNN
47.25% on CADP
[374]Srinivasan, et al.(2020) | Detection Transformer[375] | 78.2% on CADP
[376]Suzuki, et al.(2018) | QRNN+DeCAF[377] | 99.1% mAP
[357]Giannakeris, et al.(2018) | Faster RCNN | F1-score 0.33 RMSE 227 on NVIDIA CITY Track 2
[378]Arceda, et al.(2018) | YOLO+ViF[379]+SVM | 89%
[380]Biradar, et al.(2019) | YOLOv2+ CNN | F1-score 0.3838 RMSE 93.61 on NVDIA CITY Track 3
[381]Xu, et al.(2018) | Mask-RCNN+ ResCNN | F1-score 0.8649 RMSE 3.6152 on NVIDIA CITY Track 2
[382]Doshi, et al.(2020) | YOLO+KNN+K-means | F1-score 0.5926 RMSE 8.2386 on NVIDIA CITY track 4
[358]Zhou, et al.(2016) | 3D Conv | 95.2% on U-turn dataset
[359]Franklin, et al.(2020) | YOLOv3 | 100% and 95.34% for input video 1 and video 2

These pure data-driven methods often focus on detection problems.
Alternatively, other automated video-based road safety analysis frameworks
exist that use ML/DL methods to extract information about the road users from
RSU videos and perform safety analysis using domain knowledge. An important
class of such studies is extracting surrogate safety measures. For example,
the authors of [383] collected large-scale data in about 40 roundabout weaving
zones. They track the vehicles using the Kanade–Lucas–Tomasi (KLT)-based
tracker [384], then extract a set of surrogate measures, such as speed,
individual interaction measurements, and Time-To-Collision (TTC), for each
vehicle. Performing correlation analysis between the surrogate measures and
complementary data (such as design geometry and environment attributes)
reveals insightful relations about traffic safety. We also note that [385,
386] focus on the analysis of pedestrian-vehicle interactions by studying the
secondary interaction and collision risk predictions. These works are worthy
of high attention. Although these sorts of analyses require extracting more
complicated patterns from traffic videos, they can be helpful for enhancing
the real-world traffic safety. A summary of such frameworks is presented in
Table 14. We note that most of these frameworks use conventional tracking
algorithms, which yield reasonable performance. However, more advanced hybrid
detection and tracking algorithms can be developed where DL networks are used
for fast and accurate object detection, followed by a second estimator based
on the conventional methods, as discussed at the end of Section 5.3). For
instance, one may combine YOLOv5 and DeepSort [243] for precise vehicle
tracking [387].

Table 14: Automated video-based road safety analysis frameworks.

REF] authors (Year) |  | Video Process
---
Method
| Analysis
---
Method
Description
[383] St-Aubin et al. (2015) | KLT |  | Dataset Collection
[388] Zangenehpour et al. (2016) |  | KLT for tracking;
---
Bayesian classification for seg.[389];
| Regression Analysis;
---
Rank correlation analysis;
Cyclist and motor-vehicle interactions
[390] Lu et al. (2016) | KLT |  | Sensitivity analysis;
---
Golden section search
Calibrate car-following parameters in VISSIM simulation;
[391] Mohamed et al. (2018) |  | KLT;
---
Kinematic method
Motion pattern matching (MPM) by
the longest common sub-sequence (LCSS)
| Distribution/frequency analysis;
---
Clustering
| Study on impact of motion prediction methods
---
on safety indicators
[392] Xu et al. (2018) |  | YOLOv2;
---
NMS
| Database building for pedestrian analysis
[393] Battiato et al. (2018) |  |  Kernel-based object tracking [394];
---
HOG
Bayesian classification | Risk analysis from on-board video data
[385] Fu et al. (2019) |  BriskLUMINA 0.1 [395] |  | Statistic Analysis;
---
Correlation analysis
Secondary Pedestrian-vehicle interactions
[396] Xie et al. (2019) |  | KLT;
---
rPCA;
Dirichlet process Gaussian mixture model (DPGMM)
HMM | model the rear-end conflicts
[397] Chen et al. (2020) |  | HOG;
---
SVM;
Kalman Filter;
Adaptive Gaussian Mixture model
Distribution analysis |  | Safety space analysis;
---
Data collected by UAV
[398] Yang et al. (2021) |  | KLT;
---
Kalman Filter
Functional Data Analysis |  | Anomalies detection;
---
Data collected by UAV
[386] Noh et al. (2022) |  | Mask R-CNN;
---
DeepSort
LSTM | Pedestrian-vehicle collision risk prediction
[387] Chen et al. (2022) |  | YOLOv5;
---
DeepSort
| Regression Analysis;
---
Rank correlation analysis;
Coalition game theory
Traffic risk analysis

###  5.7 Sensor Information Processing

It is notable that there are several studies and datasets devoted to sensor
information analysis. Different types of commonly used sensors were provided
previously in section 3. Of particular and increasing interest is the point
cloud mapping collected from LiDARs, not just because they offer more accurate
distance measurement, but also they are considered as separate and independent
sensing of the environment to that of videos to ensure accurate perception
thus road safety. 3D point cloud mapping from LiDAR is different from grid-
based 2D images, thus different treatment strategies are pursued. LiDAR point
cloud semantic segmentation works use deep neural networks, initially treating
point cloud as construct graph [399], followed by the development of multi-
layer perceptrons to learn from raw cloud data directly [302, 303]. More
recently, the spherical projection has been employed to map LiDAR sequential
scans to depth images, and improved segmentation [400, 401, 402, 403]. Since
the focus of this paper is video-processing for traffic safety analysis, we
refer the interested readers to [404, 405, 406, 407].

###  5.8 Network-level Analysis

Traffic flow problems can be formulated as a network of mobile nodes, and
studies on individual crash analysis can be extended to the more complex setup
of network-level analysis.

There exist a few research paradigms that consider network-level analysis. One
bold example is the transportation network design and related family of
problems. Transportation network design belongs to the category of operations
research and can be divided into the Road Network Design Problem (RNDP) and
Service Network Design Problem (SNDP) according to their features and
functions[408]. The works [409, 410, 411] of RNDP aim to optimize the
performance of urban networks according to some criteria such as topology,
capacities, and flow accessibility. The works [412, 413, 414, 415, 416] of
SNDP aim to address the planning of operations for freight transportation
carriers, such as station locations, route planning, and operation frequency.

Traffic prediction studies often employ statistical techniques (such as Kalman
filtering [417], hidden Markov model [418], Bayesian interference [419]) and
DL (such as LSTM, CNN) methods [420, 421, 422, 423] to infer the network state
and produce optimal strategies for different conditions.

Network-level analysis can be used for traffic safety analysis as well. Some
key objectives would include finding correlations between traffic flow, safety
metrics geo-maps, and crash rates. For example, one may expect a direct
relation between the traffic composition (density and variety of vehicles) and
the number of crashes at different parts of highways. One may also expect
relations between the traffic flow and crash rates of nearby intersections or
segments. Network-level analysis can shed light on these highly unexplored
research areas.

There exist four general approaches for network-level analysis [424] including
traditional statistical models (e.g., [425]), endogeneity/heterogeneity models
(e.g., [426, 427]), data-driven methods [428, 429], and causal inference
models. Furthermore, we note that some models[430, 431] exploit previously
collected crash data with road information (such as Average Daily Traffic
(ADT), lane width, speed limited, shoulder width) to estimate crash
frequencies at intersections or segments. Some models [432, 433] explore risk
probabilities by processing geometric features. Recently, new models[434, 435,
436, 437, 438, 439] take advantage of advanced sensors techniques and high-
performance computation frameworks to infer real-time crash risks and take
proactive strategies. Safety analysis frameworks can perform integrative
analysis by incorporating different static and dynamic data modalities (i.e.,
imagery and sensor inputs) from different points of view to comprehend the
network status and derive realistic distributions for safety factors. An
immediate benefit of such networks would be assessing the contribution of
different factors on safety distributions and providing advisory to improve
roadways and infrastructure design as well as developing safety enforcement
and public education campaigns.

A key challenge is developing strategies and scheduling policies for data
aggregation to provide required modalities for network-level safety analysis
at the minimum cost possible. Also, data aggregation is constrained by the
utilized networking infrastructure and communication protocols between the
vehicles and roadside infrastructure. Study of such networks are out of the
scope of this work, and we refer the interested users to [440, 441, 442, 443,
444, 429, 445, 446, 440]

Recently, the ideas of using data augmentation and physics-informed neural
networks (PINN) are proposed to mimic the dynamics of complex systems while
mitigating the need for manual annotation of massive datasets. We believe that
the power of PINN and data augmentation is not yet fully utilized in this
context. There is a great potential to develop surrogate models for traffic
flow and risk analysis. The authors also believe that elegantly designed graph
neural networks can play an essential role in modeling network-level events
and trends.

##  6 Sample Problems

Safety assessment is a qualitative process, which can be approached by a set
of specific and objective sub-problems. For instance, the overall safety of a
highway section can be assessed by a set of exemplary problems provided in
Table 15. Cloud-based software can process videos captured by the roadside
infrastructure to extract statistical information from the observed events.
The results of such analyses can be used to evaluate the highway’s safety
profile and offer revisions to the traffic management guidelines to the
transportation personnel. For instance, frequent roadblocks in specific
sections of a highway may require widening the road, revisiting speed limits,
prohibiting commercial vehicles, or planning traffic re-routing.

Table 15 presents typical processing steps required for each sample problem.
As we discussed in this paper, some technical papers offer a solution for some
problems, while more work is required to solve some other problems. Also, it
is worth mentioning that some steps are necessary for each problem, while some
other pre-processing steps such as denoising and video stabilization may or
may not be included to balance between the accuracy and complexity.

Table 15: List of sample problems to assess the overall safety of a highway
segment. The short codes include VS: video stabilization, DN: denosing, SR:
super resolution, MOT: multi-object tracking, OD: object detection, IS:
instance segmentation, TE: trajectory extraction, SS: semantic segmentation,
C: classification. The step in brackets "()" denotes this step is not required
but potentially can enhance the performance. The tasks with ’*’ meaning that
these tasks can be solved by obtaining the relationships (distance, velocity)
among vehicles and determining a threshold from the relationships.

Event | Processing Steps | Examples
---|---|---
Vehicle Stopped on Shoulder | (VS)-(DN)-LD-MOT | [381]
Car Crash | (VS)-(DN)-OD/MOT-(IS)-C | [369, 364]
Emergency Vehicle on The Road | (VS)-(DN)-fine-grained OD | [183]
Careless/Evasive Lane Change | (VS)-(DN)-LD-MOT-TE-(smooting)-C | [447, 448]
Debris in Roadway | (VS)-(DN)-LD-(SR)-OD/(SS) | [310, 311, 313]
Traffic Blocked (Slowed-down) | (VS)-(DN)-MOT-(smoothing)-TE-(SE) | [449, 450]
Sharp Braking | (VS)-(DN)-MOT-(smoothing)-TE | [451]
Passing Red Traffic Light | (VS)-(DN)-(OD for ligthts)-MOT | [452]
Traffic Composition | (VS)-(DN)-OD/MOT | [453]
*Vehicle Driving Obviously Excessively Slow | (VS)-(DN)-MOT-TE-(smoothing) | -
*Vehicle Driving Obviously Excessively Fast | (VS)-(DN)-MOT-TE-(smoothing) | -
Vehicle Driving in the Prohibited Area | (VS)-(DN)-LD-OD | [454]
Invalid Car in HOV | (VS)-(DN)-LD-OD | -
Improper (Careless) Entering the Road | (VS)-(DN)-LD-MOT-TE-(smooting)-C | [455]
Improper (Careless) Exiting the Road | (VS)-(DN)-LD-MOT-TE-(smooting)-C | [455]
Zigzag Driving | (VS)-(LD)-MOT-TE-(smooting)-C | [447]
*Distance Violation to Front Vehicle | (VS)-(DN)-MOT-(smoothing)-C | -
*Distance Violation to Side Vehicle | (VS)-(DN)-MOT-(smoothing)-C | -
Violation of The Lines | (VS)-(DN)-LD-MOT-(smoothing)-TE | [454]
Pedestrian on The Road | (VS)-(DN)-OD | -
Bike/Motorcycle on The Road | (VS)-(DN)-OD | -
Trailer [Oversized Car] on The Road | (VS)-(DN)-OD | -

Here, we provide alternative processing pipelines for zigzag driving as an
exemplary application, along with a comparative analysis on the accuracy and
complexity of each method. Zigzag driving on highways is a single-vehicle-
involved event defined as frequent shifts to left and right in a short period
of time $t_{s}$. Figs. 14(c)-(e) show the three bases of zigzag driving during
the short period $t_{s}$, in comparison with the normal driving (including the
normal lane change shown in Fig. 14(a), and the normal straight driving shown
in Fig 14(b)). These zigzag driving incidents include i) zigzag driving only
once without lane change (Fig. 14(c)). ii) zigzag driving once with lane
change (Fig. 14(d)), and iii) zigzag driving once with multiple lane changes
(Fig. 14(e)). The other cases of zigzag driving shown in Figs. 14(f)-(h) can
be decomposed into three base cases. To solve such an easy problem, one may
take different approaches, including:

Figure 14: (a)-(d) are the trajectories with normal driving. (a) shows the
normal lane changing, and (b) shows the normal straight driving. (c)-(e) are
the three bases of zigzag driving. (c) is zigzag driving once without lane
changing. (d) is zigzag driving once with lane changing once. (g) is zigzag
driving once with lane changing twice. (f)-(h) are some examples of more
aggressive and frequent zigzag driving. The blue circles in (h) denote the
sampling points by HQ-sampling with a big interval $t_{0}$

  1. 1.

MOT-TP: This solution uses Multi-Object Tracking (MOT) to extract cars’ motion
trajectories. The overall process is as follows: Several sub-sequences of each
trajectory are generated by using a sliding window with size $t_{s}$ (in order
to save processing time, the stride can be set greater than 1). Then a sub-
window with size $L$ is used to extract possible peaks and trough points to be
processed by Non-Maximum Suppression (NMS) to obtain the turning points. When
the number of the turning points is greater than the threshold $n_{z}$, this
sequence will be determined as zigzag driving. Theoretically, MOT-TP is able
to detect all cases of zigzag driving, regardless of the road pattern and
without the need for detecting road lanes. The potential drawback of this
method is that normal driving on sharp road curves may be considered zigzag
driving.

  2. 2.

LD-Markings: First, the lanes are detected by instance segmentation. Then,
similar to the MOT-TP method, the motion trajectories are extracted by MOT.
The zigzag driving is determined by comparing the trajectory sub-sequences
with the lane markings. The trajectory is determined as zigzag driving when
the number of lane changes is greater than a threshold $n_{z}$ within a given
time interval. The accuracy of this method varies based on the performance of
the utilized algorithms for trajectory extraction and lane detection. It is
not suitable for dirt and unlaned roads. It also misses zigzag driving with no
lane changes, such as the examples in Figs. 14(f,h).

  3. 3.

Trj-Cls: The Trajectory classification method skips modeling the road and the
surrounding environment; instead, it directly applies supervised or
unsupervised classification to the extracted trajectories to identify zigzag
driving events. Similar to the LD-markings, the accuracy of this method
depends on the accuracy of MOT used for trajectory extraction. It is flexible
and can utilize different classification methods. However, this data-driven
method requires a relatively large manually annotated dataset to maintain
reasonably high classification accuracy.

  4. 4.

HQ-sampling: This method can be viewed as a simplified version of LD-Markings.
It uses high-resolution data to accurately identify the plate license numbers
(by Optical Character Recognition (OCR)). This solution can track each vehicle
by employing plate license numbers to replace the tracking algorithms. Instead
of using video, this high-quality data often is in the form of images, to be
captured continuously at intervals $t_{0}$, which may lead to inaccurate
trajectory extraction. Similar to the LD-markings method, HQ-sampling
determines zigzag driving by comparing the detected number of lane changes
with the threshold $n_{z}$. Its accuracy depends on the performance of the
OCR, the lane detection algorithm, and the utilized sampling interval $t_{0}$.
The main shortcoming of this method is high complexity for short intervals,
compromised accuracy for long intervals. This method may also undercount the
lane changes if the sampling interval is selected large (e.g., it only counts
zero lane changes in Fig 14(h) with the three sample points).

Table 16: Comparison of the solutions to detect zigzag driving.
SW denotes sliding window used to search the peak and trough points.

Method |  | Complexity of
---
Inference
| Sources of
---
Complexity
Accuracy
MOT-TP | M | MOT+SW | H
LD-markings | L | LD+MOT | M
Trj-Cls | H | MOT+ML | H
HQ-sampling | M/H | LD+OCR | L/M

Table 16 compares these four solutions in terms of accuracy and computation
complexity. Note that the comparison is based on the assumption that the MOT
algorithm performs well and a powerful trajectory smoothing method is
deployed. Additionally, roadside video cameras are often fixed, meaning that
lane detection is performed only once per scene and does not substantially
affect the computation load.

As seen, there exist many alternative solutions for such a simple problem.
Therefore, a thorough understanding of methods can help the researchers design
the most effective processing pipeline for the problem at hand. It is worth
mentioning that other trajectory-based tasks often have similar processing
steps of (VS)-(DN)-LD-MOT-TE-(smoothing)-C, as shown in Table 15. It means
that some parts of processing pipelines can be shared among different
problems, or transferable inference models can be used for multiple tasks to
lower the cost.

##  7 New Trends in Deep Learning

In this section, we review new trends in deep learning that are expected to
substantially influence the field of video-based traffic safety analysis.

###  7.1 Computer Vision by Transformers

We reviewed various DL methods for vision-based traffic video analysis in
Section 5. Most DL networks include Fully Connected (FC) layers, Convolutional
Neural Networks (CNN), or Recurrent Neural Networks (RNNs) as their backbone.
This journey witnessed several milestones such as the emerge of LeNet [456] in
1989, AlexNet [110] in 2012, VGG-Net [111] in 2014, and GoogleNet[112] in
2015. ResNet [113] was amongst the most surprising developments that beat
Human perception with a 3.57% top-5 error rate in the ImageNet competition.
Parallel to CV, DL methods are developed for sequential learning, mainly in
the field of Natural Language Processing (NLP). For years, sequential learning
was dominated by Gated Recurrent Units (GRUs) [457] and Long Short-term Memory
network (LSTM) [458], until recently when a revolutionary framework known as
Transformer is introduced by Google researchers [459]. It has an encoder-
decoder architecture and uses multi-head self-attention modules to capture
longer internal dependencies in addition to the input-output dependencies in
sequential data. Position embedding enables parallelizing the training process
and capturing dependencies beyond the sequential relations. Technical details
and different variants of transformers are discussed in [459, 460].

Transformers are proven to be successful in capturing short-term and long-term
dependencies in both NLP and CV tasks. The most classic work that uses
Transformer architecture is [461] from Google, which outperforms the
competitors in 11 NLP tasks. The emerge of Transformers is considered the end
of the LSTM era in NLP, and can even challenge the position of CNN in CV tasks
in the next few years. For instance, Google’s implementation of Transformers
[462] achieved an unprecedented accuracy of 88.55% on ImageNet, taking
advantage of transfer learning. They pre-trained the network on a very large
dataset (e.g., JFT-300M), then fine-tuned if for ImageNet. Similarly, the
authors of [463] achieved an 83.3% top-1 accuracy and beat the ResNet50 with a
similar number of parameters. They showed the Transformers’ capability on
downstream tasks, such as detection and semantic segmentation. We expect to
see more Transformer-based safety analysis frameworks in years to come, not
only in CV tasks, but also for processing sequential data, such as trajectory
extraction, tracing, and individual and collective behavior modeling of
pedestrians, self-driving cars, and AVs.

###  7.2 Federated Learning

One of the challenges in traffic safety analysis is processing massive volumes
of RSU videos. Several approaches, such as cloud computing, edge/fog
computing, down-sampling, resolution reduction, importance sampling, and
event-triggered on-demand processing methods, are developed by researchers to
tackle this problem while not missing key safety events. One key research
direction is developing optimal strategies for data and model sharing so that
more meaningful information is extracted by collective processing of the
aggregated data in RSUs. One of these methods is using Federated Learning
(FL).

Federated learning was first introduced by Google [464] in 2017 as a
decentralized collaborative technique. Federated learning aims to address
several practical issues in practical scenarios, such as vehicular edge
computing. Some of these issues include: i) the data is often in silos for
privacy problems; ii) the communication capacity is limited; iii) data is
unbalanced in different local nodes, and iv) data not be independent and
identically distributed (non-i.i.d), for example, in-board cameras in two
vehicles from different regions can both record specific types of roads, but
the roads in different regions often have different styles. Federated learning
is categorized into three classes based on data partitioning [465]: i)
horizontal federated learning, which has more overlapped features with fewer
overlapped users; ii) vertical federated learning, which has less overlapped
features but with more overlapped users; and iii) federated transfer learning,
with non-overlapping features and users. To train an FL model, the local nodes
only train on their local data and upload the encrypted learned weights or
gradients to the central server without sharing data. Then the central server
aggregates this information to update the global model and release the model
to the nodes for the next round of learning. Two baselines algorithms, known
as FedSGD and FedAvg, are implemented in [464]. Specifically, in each round of
FedSGD, each local client updates once, and then the global model is updated
based on the weighted average of the updated gradient from the clients. A
little different from FedSGD, FedAvg performs more aggressively, in which the
local client takes more steps for each central server update. The experimental
result shows that FedAvg imposes less communication load compared to FedSGD to
achieve the same accuracy. The authors of [466] proved the convergence of
FedAvg under strong convexity and smoothness assumptions. In past years, some
works have already used FL in smart transportation, such as a privacy-
preserving traffic prediction introduced in [467, 468], and FL-based Vehicle-
to-Vehicle (V2V) and Vehicle-To-Everything (V2X) low-latency communication
introduced in [469, 470]. We refer the interested reader to [471] for a more
interesting and broader range of applications of FL.

It is noteworthy that FL still has some unresolved issues. Firstly, its
reliability and robustness under uncertain conditions are not yet well
established. Specifically, as discussed before, the center server has no
access to the entirety of the local data and is unaware of local update
progress. It means the aggregated model is much vulnerable to defending the
introduced backdoor functionality if one or some nodes are malicious [472].
Additionally, most practical FL networks consist of heterogeneous local nodes
with varying learning ability and communication rates, which can challenge the
overall robustness and performance of the system. After solving its issues, FL
would be a promising solution for traffic safety analysis, as it provides a
powerful tool to extract and analyze massive data volumes while protecting the
privacy of road users. RSUs and in-vehicle processors can act as central and
local processors of a large-scale FL system.

###  7.3 Adversarial Learning For Safety Analysis

Despite the demonstrated power of DL methods in learning from massive data,
the community is increasingly concerned about the security and reliability of
neural networks. One reason for this concern is that small perturbations can
easily fool a big DL network [473], which can lead to misleading results in
traffic safety analysis and operation, especially for self-driving cars. Fig.
15 demonstrates an example that the adversarial sample generated by Fast
Gradient Signed Method (FGSM) [474], which easily fools MobileNetV2 to detect
a traffic light as a fence.

Figure 15: An example of adversarial samples generated by Fast Gradient Signed
Method (FGSM) [474]. The detector is MobileNetV2 trained on ImageNet. (a) is
the normal sample, classified as "traffic light" by MobileNetV2, (b) is the
hand crafted noise (adversarial attack), and (c) is the adversarial sample
generated by adding hand crafted noise to the normal sample, classified as
"fence" by MobileNetV2.

Adversarial machine learning aims to alleviate these issues by exploring the
vulnerabilities of ML algorithms and defending against potential attacks. The
attacks can be divided into two main categories: evasion/exploratory attacks
and Poisoning/causative attacks [473, 475]. The former cannot tamper with the
model but feed carefully designed adversarially perturbed examples to the
network in the testing phase. These data (sometimes imperceptible) can
significantly change the feature representations and result in fatal
misclassifications. In the latter case, the attacker injects malicious data
samples into the training dataset to misguide the model training. It is a
legitimate concern if one anticipates that adversarial samples are inserted in
the training dataset using software backdoors [476]. Alternatively, one way
poison the training process is by explicitly attempting to corrupt the trained
model. For instance, poisoning data samples can be added to cause gradient
vanishing [477].

Indeed, one would suppose that the attacker and defender play a two-player
game, where the defender expects to design a robust model against the
adversarial samples. It augments the data by adding the adversarial samples in
the training phase, and then solves a min-max problem. Another strategy is to
try to recognize and filter out the adversarial examples [478]. Since many
attacks are implemented by tracking the gradient, some defenders (such as
[479]) attend to hide or change the gradient information. Some other defenders
design robust optimization based on regularization methods against attacks
[480].

Finally, we remind that these attacks or defense methods are often designed
for specific or known situations. However, real-world applications are often
exposed to unknown attacks. This concern can be even more critical when
developing traffic-related frameworks. The driving safety analysis requires
further investments in developing robust models using inverse learning. The
models should undergo heavy inspections before being utilized in traffic
control systems.

###  7.4 Meta-Learning

Meta-Learning, aka learn-to-learn, aims to learn the learning process from
other machine learning or deep learning tasks, and then apply this learned
experience to guide the design and training process for a new task. It can
generally learn the new task faster than the manual training; therefore
facilitates designing a novel algorithm or framework merely by data-driven
methods [481]. To be more specific, in general learning, the optimizer only
learns on the current dataset, and the objective function is defined only for
the current task. However, meta-learning focuses on the elements of the
learning phase (such as parameter initialization, optimizer, architecture,
etc.) and learns how to generalize these elements across tasks and how to
perform better on new tasks. Formally, the problem is presented by [482] as:

$\displaystyle\underset{\omega}{\min}\sum\mathcal{L}^{\text{meta
}}\left(\theta^{*(i)}(\omega),\omega,\mathcal{D}^{(i)}\right),$ (1)
$\displaystyle\text{ s.t.~{}~{}~{}
}\theta^{*(i)}(\omega)=\arg\underset{\theta}{\min}\mathcal{L}^{\text{task
}}\left(\theta,\omega,\mathcal{D}^{(i)}\right),$

where $\omega$ denotes the meta prior to be learned, $\theta^{*(i)}(\omega)$
denotes the learned parameters on the dataset $\mathcal{D}^{(i)}$ for task $i$
under the meta prior $\omega$. $\mathcal{L}^{\text{meta }}$ and
$\mathcal{L}^{\text{task }}$ denote the loss of meta-learning and the task
itself, respectively. Meta-learning can be trained using a non-parametric
setup by measuring the distance between the target and trained samples, and
predicting the sample label based on the closest trained samples. A
differentiable $\mathcal{L}^{\text{meta}}$ can be solved by gradient descent.

It is noteworthy that this concept is somewhat related to transfer learning
but is also slightly different from it. Specifically, transfer learning aims
to learn the regular and trainable parameters, such as the weights of the
networks from other datasets or tasks, then fine-tunes the pre-trained model
for a new task, while meta-learning tries to learn the hyper-parameters from
other tasks and then utilizes them to learn a new task from scratch. Both
transfer learning and meta learning can help develop more accurate driving
safety analysis, taking advantage of similar datasets.

###  7.5 Unsupervised Spatiotemporal Representation Learning

Video data plays an essential role in traffic-related tasks; however,
supervised learning can be prohibitively expensive in some traffic scenarios,
such as driving anomaly detection, due to the high cost of data annotation.
Under such limitations, unsupervised and self-supervised learning methods
would be an appealing way of processing traffic video.

One approach to developing unsupervised learning is taking advantage of
representation learning to translate raw input into more workable
representations. An exemplary scenario would be representing video frames in a
latent space that shows a substantial distinction between safe and unsafe
traffic volumes.

Generally, the goal of unsupervised learning is learning data in a latent and
often a lower-dimensional space, which offers temporal invariance and
simplifies the downstream analysis. In 2021, He et al. [483] investigated the
performance of four popular image-based unsupervised frameworks (Momentum
Contrast (MoCo) [484], a Simple framework for Contrastive Learning of visual
Representations (SimCLR) [485], Bootstrap Your Own Latent (BYOL) [486], and
Swapping Assignments between multiple Views of the Same image (SwAV) [487])
for the spatiotemporal representation learning. The authors named the
spatiotemporal invariant as temporally-persistent. Specifically, they
anticipated that the high-level representation of the same video would be
similar in different clips. These four frameworks take advantage of
contrastive learning, which aims to maximize the similarity of positive
samples and minimize the similarity of negative samples. Generally, the
negative samples are the sampled clips from other videos. In MoCo, the authors
introduced a creative method known as momentum coding, which constructs a
dynamic dictionary to manage the representation of samples. This allows to
quickly build and update the large and consistent dictionary to store much
representation with a lower memory cost, which hugely enhances the ability of
contrastive learning. BYOL and SwAV are considered versions of SimCLR and MoCo
without using negative samples, respectively. They showed that these extended
image-based frameworks could learn the spatiotemporal representation of video.
A more interesting result is the competitive performance of these frameworks
when combined with downstream tasks, compared to supervised learning in many
tasks. These results provide a limitless possibility to deploy the
unsupervised frameworks to industry and real applications.

##  8 Traffic Datasets

Figure 16: The visualization of some exemplary crash data. Each point in
(a)-(c) denotes a crash, and (d)-(f) are the corresponding heatmaps. Table 17:
Some popular traffic-related datasets.

Category | Dataset | Size | Features
---|---|---|---
Trajectory analysis | MIT traffic dataset(2011)[123] |  | around 162, 000 frames
---
from a 90 minutes long video sequence (30 fps)
| Both pedestrian and vehicle movements;
---
with occlusions and varying illumination conditions
MIT trajectory dataset(2008)[488] | 577 radar tracks, 45 453 video tracks | Multiple single camera view
CVRR trajectory analysis datasets (2011) [124] |  | 4 scenes: a simulated intersection (19 activities),
---
a real highway (8 activities)
| Units are pixels;
---
only contain spatial information
i-Lids–advanced vehicle detection challenge[489] | 35000 images. 7 sequences with 25 fps. |  | Consider daytime;
---
various locations in the UK
NGSIM(2007)[490] | 45 mins+45 mins+ 30 mins |  | Three sub-datasets;
---
US-101 and I-80 for freeway scenario;
Lankershim Boulevard Dataset for arterial corridors scenario;
each dataset is segmented with different congestion conditions
QMUL junction dataset(2012) [125] | 1 hour video | Busy traffic at a junction
Auto-driving tasks | CamVid(2009) [491] | 700 images, 32 classes, 10 min HD video with 30fps |  | Support pixel-wise segmentation tasks;
---
provide 3D camera pose
Cityscapes (2016)[492] |  | 5000 pixel-level annotated images
---
and 20000 coarse annotated images (weak-label)
| complex urban street scenes;
---
record from 50 cities;
Mapillary Vistas Dataset(2017)[493] |  | 25 000 high-resolution images;
---
66 classes for objection
37 instance-specific labels
| Various conditions regarding weather,
---
season and daytime
SYNTHIA(2016) [494] |  | 13,400 frames of the city (real);
---
four video sequences of 50,000 frames (simulated);
13 classes
| Scene diversity;
---
multiple seasons;
lighting conditions and weather
KITTI Vision Benchmark Suite (2012)[164] | 7481 training images and 7518 test images; |  | Up to 15 cars and 30 pedestrians per image;
---
tasks of interest are stereo, optical flow,
visual odometry, 3D object detection, and 3D tracking
Berkeley Deep Drive (2018)[495] | over 100K videos |  | Diversity of annotations;
---
diversity of geography, environment, and weather
IDD (2019) [335] | 10,004 images; 34 classes; 182 drive sequences |  | Unstructured environments;
---
four-level label hierarchy; possibility for new tasks:
domain adaptation,
few-shot learning and behavior prediction
Traffic sign detection | TLR (2009)[496] |  | 11 179 frames (8min 49sec, @25FPS);
---
9 168 instances of traffic lights
| Hand-labeled;
---
tasks for traffic light detection and traffic sign detection
LISA (2012)[497] |  | 6600 frames with 7800 annotated objects;
---
49 classes of signs
| Recorded in the US;
---
Images from various camera types
GTSDB(2013)[498] |  | 900 images
---
divided into 600 training images (846 traffic signs)
and 300 evaluation images (360 traffic signs)
| Signs are divided into prohibitory, danger, mandatory,
---
and other signs
BelgianTS(2014)[499] |  | 13,444 traffic sign annotations in 9,006 still images
---
corresponding to 4,565
physically distinct traffic signs.
Available for 2D and 3D testing
TT100K (2016)[215] |  | 100000 images containing 30000 traffic-sign instances;
---
128 sign classes
Diversity of annotation
BSTL (2017)[500] |  | 13427 images (5093training+8334test 15fps);
---
24000 annotated traffic lights
| Cover a decent variety of road scenes and
---
typical difficulties
Plate detection | ReId (2017)[501] | 9.5hours from 8 locations |  | Varied daytime;
---
multi-plates on a frame;
cameras placed on bridges above highways;
relatively small objects
UCSD car dataset(2007) [502] |  | 10 hours video;
---
878 cars tracked from the moment;
291 still cars recorded in parking lot
Label with make, model, license plate location, plate texts
UFPR-ALPR Dataset(2018)[503] |  | 4,500 fully annotated images from 150 vehicles;
---
30 fps
>30,000 license characters;
| Different license color;
---
both vehicle and camera are moving
SSIG Dataset(2016)[504] |  | 2000 Brazilian license plates;
---
14,000 alphanumeric symbols
| Static camera view;
---
character are segmented

Table 18: Some popular traffic-related datasets (continue).

Category | Dataset | Size | Features
---|---|---|---
Naturalistic Study | The 100-Car Naturalistic Driving Study(2006)[505] |  | 2,000,000 vehicle miles;
---
43,000 hours of data
| Contains extreme cases of driving behavior;
---
Event-based database
SHRP 2 NDS dataset(2012)[506] | more than 2 PB of continuous naturalistic driving data |  | Multi-view video outputs;
---
collected during a 3-y period from
more than 3,500 participants, aged 16–98
UAV | MOR-UAV(2020)[507] |  | 89,783 moving object instances;
---
10,948 various scenarios;
| Tasks for moving and non-moving objects;
---
varied conditions of the environment;
labeled by aixs-aligned bounding boxes
Stanford Drone Dataset(2016)[508] |  | >100 top-view scenes;
---
20,000 targets
annotated trajectories and id
| Real-world scenario-based;
---
varied types of targets
The highD dataset(2018)[509] |  | 16.5hous of measurements for 110,000 vehicles from 6
---
locations.
| Real-world scenario-based;
---
focus on highways
contain naturalistic behavior of road users
The inD dataset(2019)[510] |  | contains more than 11500 road users;
---
10 hours measurement
| Successor of highD,
---
focus on intersections
Video-based Traffic events (Crash) | UCF-Crimes(2018) [511] | 1900 sequences, 128hours |  | Includes realistic anomalies such as fighting, road crash;
---
No spatial annotation
DAD(2016) [373] | 1730 sequences, 2.4 hours |  | Spatio-temporal annotation
---
Video captured by dashcam
CADP(2018)[372] | 1416 sequences, 5.2 hours |  | Include multi-crash sequences
---
Captured by CCTV; Include varied environment
NVIDIA AI CITY challenge[512] |  | Track 1: 31 videos (about 9 hours in total)
---
Track 2: 56,277 images,
36,935 come from 333 object identities
Track 3: 215.03 minutes videos
300K bounding boxes
Track 4: 200 15mins videos 30fps
| Track 1: vehicle counting
---
Track 2: vehicle Re-id
Track 3: vehicle tracking
Track 4: Abnormal detection
Anomalies can be due to
car crashes or stalled vehicles
Fine-grained Vehicle Classification | Standford Cars[78] | 16,185 images of 196 classes of cars |  | The cars images are taken from many angles;
---
classes typically based on Make, Model, Year
COMPCARS[79] |  | Web-nature: 136,726 (entire car)+27,618 (car parts),
---
163 car makes with 1,716 car models;
Surveillance-nature: 5000 car images
| Web-nature data are labeled with
---
bounding boxes and viewpoints;
the surveillance-nature images are in the
front view
Pedestrian Detection | Caltech [163] (2009) |  | 61k images with 192k pedestrians to train;
---
56k images with 155k pedestrians to test;
1273 unique pedestrians
| Objects with different occlusion;
---
Objects with multi-scale;
Sequential images
CityPersons [513] (2016) |  | 5k images with 35k pedestrians;
---
19654 unique pedestrians
| Built upon Cityscapes;
---
Record in different city with different weather;
Objects with different occlusion;
KAIST [165] (2015) |  | 95,328 annotated image pairs (thermal and visible),
---
103,128 pedestrian annotation,
1182 unique pedestrians
| Multi-spectral;
---
Include partial occlusion and heavy occlusion;
Include day and night;
Sequential images
CVC-14 [172] (2016) |  | 5051 grayscale visible and thermal frame pairs;
---
9,319 pedestrian annotation (visible);
8,750 pedestrian annotation (thermal);
| Multi-spectral;
---
Include day and night;
Sequential images;
Annotation include mandatory and unclear persons
FLIR [514] ($\sim$2018) | 9,711 thermal and 9,233 RGB images ; |  | Multi-spectral;
---
15 Classes;
Sequential images;
Many pairs are unaligned;

Table 17 and Table 18 provide a relatively complete list of datasets that can
be used for different aspects of video-based analysis for different CV-based
traffic tasks. Data for trajectory analysis is often extracted from still
cameras or drone-captured videos and used to understand microscopic-level
(such as car-following models) to macroscopic-level traffic flow (such as
traffic wave models) [515]. Safety analysis of human-driven vehicles closely
relates to video understanding tasks such as scene recognition, object
detection, instance and semantic segmentation, developed for auto-driving
vehicles. Traffic sign detection is another difficult problem due to using
different variants for the same signs, uncontrolled and varying light
conditions, the impact of weather conditions (rain, snow, fog, etc.), and the
unpredictable traffic environment, especially when real-time processing is
needed.

Naturalistic Driving Studies often record and process the driver’s behavior,
cognition, and perception of the surrounding environment without influencing
or distracting the driver. This kind of data is generally large-scale since it
requires long-term observations with multiple observed objects. The plate
detection task involves drawing a bounding box around every detectable plate.
A sub-task of plate detection is Optical Character Recognition (OCR), which
requires understanding and validating engraved license plate numbers. Another
twist to this problem is the angle of view. While most images are taken by
roadside cameras, drone-based imagery datasets provide top-view images that
solve some problems but poses new image processing challenges like human
detection and small object detection.

It is notable that general Computer Vision (CV) datasets, presented in Table
19, can be used to approximately evaluate the performance of developed
algorithms if the desired traffic-specific dataset is not readily available.
Here, we also want to mention some popular CV datasets (shown in Table 19)
since the most recent state-of-the-art algorithms deploy them as their
evaluation criteria.

Crash Data: Police-reported Crash Data, generally collected and stored by
local, regional, state, and/or national government agencies after traffic
crashes, can provide official, objective information about crash incidents.
Sanitized crash data (e.g., personal information removed) is generally
publicly available at the state level either online (e.g., Michigan crash data
can be obtained at [516]) or by a public records request. National level crash
data in the US is available from the National Highway Traffic Safety
Administration (NHTSA) via the National Automotive Sampling System General
Estimates System (NASS-GES)[517] which provides a national sample of police-
reported crashes and the Fatality Analysis Reporting System (FARS)[518] which
provides details on nationwide fatal crashes. Crash data sets provide a wealth
of information related to crash circumstances (including sequence of crash
events and crash type), environmental conditions at the time of the crash,
roadway characteristics at the crash scene, vehicle/road user information, and
crash involved person-level characteristics (e.g., injury severity, age,
gender, impairment, etc.…), among other information. The Model Minimum Uniform
Crash Criteria (MMUCC)[519] provides a voluntary guideline for agencies with
respect to the minimum data elements that should be included in crash
databases and includes a description of 115 recommended data elements related
to the incident, vehicle, person, roadway, and other categories. Crash data
can be used to revise traffic and/or roadway plans, develop safety
countermeasures, and explore associations between crashes and traffic safety
violations or other non-crash safety metrics. It can also be used to
incorporate prior knowledge when analyzing traffic safety events.

Missing Datasets: There is a critical need to develop new datasets for traffic
analysis that cover the underexamined aspects. As mentioned later in section
10.1, some studies consider driving safety from the behavioral science
perspective. For instance, eye motion tracking can be used to gauge the
driver’s attention and detect distraction episodes [520, 521, 522, 523, 524].
Very few datasets exist to facilitate such research. Among these, we found two
datasets (The 100-Car Naturalistic Driving Study[505] and SHRP 2 NDS
dataset[506]) that offered a relatively large number of samples for driver
behavior analysis. However, we noted that in these datasets, the activities
for one vehicle in a specific scene are non-repeating, meaning that these
datasets cannot provide driver-specific information. The authors of this study
are currently working to develop a small dataset for driver-specific anomaly
detection by collecting aerial imagery from test drivers’ behavior on specific
scenarios multiple times. This dataset would enable profiling drivers based on
their reaction to traffic conditions and use it to find abnormal behaviors
that can be indicators of driving issues and potential crash risks. Also, very
few datasets record traffic events from different perspectives. Datasets that
can offer roadside imagery, along with aerial imagery and car-mounted cameras
for synchronous analysis of traffic views, can open new research directions,
particularly for multi-modal video analysis.The visualization of some
exemplary crash data from Arizona Department of Transportation (ADOT) is
presented in Fig.16.

Table 19: Some popular CV datasets.

Dataset | Size | Features
---|---|---
MS-COCO[525] | 91 objects types. 2.5 million labeled instances in 328k images. |  | A set of datasets.
---
labeled by per-instance segmentation;
more instances per category
PASCAL VOC[188] | 20 classes. 11530 images. |  | A set of datasets.
---
Provide standard evaluation procedures;
contain significant variability in terms of object size,
orientation, pose, illumination, position and occlusion
ImageNet[526] |  | 21841 non-empty synsets, 14,197,122 images.
---
1,034,908 mages with bounding box annotations.
Organized according to the WordNet hierarchy.

##  9 Safety Metrics

An essential objective of driving safety analysis is extracting operational
safety metrics, which are quantifiable measures extracted from traffic videos
(or other data sources) that determine the relative risk of an event that may
lead to a crash. Some important safety metrics that are used to analyze car
crashes include:

  1. 1.

Temporal-based indicators: Time to Collision (TTC), Extended Time to Collision
(Time Exposed Time-to-Collision (TET), Time Integrated Time-to-
Collision(TIT)[527]), Modified TTC (MTTC), Crash Index (CI), Time-to-Accident
(TA), Time Headway (H), and Post-Encroachment Time (PET).

  2. 2.