# Cooperative Saliency-based Obstacle Detection and AR Rendering for Increased
Situational Awareness

Gerasimos Arvanitis, Nikolaos Stagakis, Evangelia I. Zacharaki, and
Konstantinos Moustakas G. Arvanitis, N. Stagakis, E. I. Zacharaki and K.
Moustakas are with the Department of Electrical and Computer Engineering,
University of Patras, Greece (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Autonomous vehicles are expected to operate safely in real-life road
conditions in the next years. Nevertheless, unanticipated events such as the
existence of unexpected objects in the range of the road, can put safety at
risk. The advancement of sensing and communication technologies and Internet
of Things may facilitate the recognition of hazardous situations and
information exchange in a cooperative driving scheme, providing new
opportunities for the increase of collaborative situational awareness. Safe
and unobtrusive visualization of the obtained information may nowadays be
enabled through the adoption of novel Augmented Reality (AR) interfaces in the
form of windshields. Motivated by these technological opportunities, we
propose in this work a saliency-based distributed, cooperative obstacle
detection and rendering scheme for increasing the driver’s situational
awareness through (i) automated obstacle detection, (ii) AR visualization and
(iii) information sharing (upcoming potential dangers) with other connected
vehicles or road infrastructure. An extensive evaluation study using a variety
of real datasets for pothole detection showed that the proposed method
provides favorable results and features compared to other recent and relevant
approaches.

###### Index Terms:

pothole detection, collaborative awareness, point cloud processing, augmented
reality, CARLA, visualization, driver’s safety

##  I Introduction

Information-centric technologies have started to play a central role in the
recent automotive industry boosting new research trends in semi or fully
Automated Driving Systems (ADS). Autonomous vehicles, ranging from level 3 to
level 5 of autonomy [1], are expected to operate safely in real-life road
conditions, but the reality is that obstacles like potholes, bumps, and other
unexpected objects are not uncommon in an everyday driving context. For this
reason, the detection and identification of obstacles are imperative for
reliable operation of autonomous vehicles [2].

Moreover, driver inattentiveness plays a major role in driving safety and is
the culprit of road accidents around the world [3, 4], thus a lot of work has
been devoted in the quantification of the abstract mechanics of human
situational awareness [5]. Enhancing situational awareness is especially
critical in the case of semi-autonomous cars, where the operator may be
distracted by secondary activities, e.g. looking at the phone or reading a
book. If the driver has to take over control, it is important to minimize the
required reaction time. This can be achieved by monitoring and presenting to
the driver the crucial information about the environment, thus keeping him/her
aware of potentially hazardous situations. Inherent challenges include the
need for unobtrusive information display, avoiding the effects of tunnel
vision which could lead to actually overlooking critical information [6].

The problem of road pothole detection is commonly targeted using imaging
(camera) data and computer vision techniques [7, 8, 9]. Although image-based
techniques have achieved great success, one common drawback is that they are
sensitive to motion blur and changes in lighting and/or even shadows [10].
Also, most techniques do not account for other passing vehicles [11]. This can
make them unreliable in real use cases, which is a major weakness in problems
involving human safety. In light of all this, the use of a 3D LiDAR (Light
Detection and Ranging) sensor could provide more robust sensing capabilities
for the analysis of potholes, in the same way that it is used to increase the
accuracy of road’s boundary detection [12, 13, 14]. On the other hand, a
limitation of the LiDAR sensor is that, due to refraction and reflection,
water appears as a black hole in the imagery calculated from LiDAR data [15],
imposing additional challenges in the detection of potholes filled with water.

The purpose of this work is to increase the driver’s situational awareness
through automated cooperative obstacle detection, visualization and
information sharing with other connected vehicles in a V2X (vehicle-to-
everything) setting. To address the above issues, we developed a point cloud
processing system that takes as input road environment data and classifies
them into safe and potentially hazardous regions by identifying obstacles
lying in the range of the road. We selected LiDAR as sensing modality for the
surrounding environment due to its ability to retrieve depth information and
its large range, making it suitable for driving environments. For more robust
estimation, LiDAR data are fused with information on driving patterns, such as
the steering angle of the wheels. For implementation and evaluation, we
utilized the open-source CARLA simulator [16] including also a multi-agent
system of vehicles, and we augmented it with our obstacle detection and
tracking component. In this simulated environment, information sharing between
agents is enabled, so that vehicles are notified about incoming obstacles even
when there is no direct line-of-sight.

To avoid any information visualization clutter, we propose the use of AR for
visualizing critical information in the driver’s field of view. AR rendering
is based on classical perspective projection, where for each point (of the
point cloud) the pixel coordinates in the image space of the AR interface are
calculated through projection and a color is assigned indicating the object
class. Interfaces that can be used for in-vehicle visualization include AR
headset, Head-Up Display (HUD) [17, 18] or even the car’s windshield with
transparent display.

The contributions of the proposed approach can be summarized as follows.

  * •

Development of an obstacle detection module that takes into account the
extraction of saliency maps from point clouds.

  * •

Generation of data for randomized multi-ego connected vehicle in cooperative
driving scenarios.

  * •

Creation of realistic synthetic data of potholes that can be entered in the
town maps of the CARLA simulator for the design of lifelike driving
situations.

  * •

AR visualization for point cloud projection registered on the scene images.

  * •

Development of public and open access libraries with code for the
aforementioned components111https://github.com/Stagakis/saliency-from-
pointcloud,222https://github.com/Stagakis/carla-data-
generation,333https://github.com/Stagakis/roadpatch-with-pothole-
generator,444https://github.com/Stagakis/carlapclprocessing.

The rest of this paper is organized as follows. First we present previous
works in related domains in Section II, and then describe in detail the
proposed methodology in Sections III and IV. Section V follows with some
experimental results in comparison with other state-of-the-art methods, while
Section VI draws the conclusions and directions for future work.

##  II Previous Work

In the following we provide an overview of methodologies tackling the main
challenges of the presented approach on (i) obstacle detection, (ii)
cooperative driving and (iii) AR infotainment systems.

####  1 Obstacle Detection

A major element that adds unpredictability in path planning for self-driving
cars are obstacles in the road. Obstacles can appear in the form of objects
beyond the surface of the road, or cracks and holes in paved areas. There has
been major work on obstacle detection, raging from real-time implementations
[19], to offline schemes that act as automated informants to the authorities
responsible for maintenance [20], or as efficient unsupervised techniques for
pothole detection [21]. Most of the existing works implement a broad spectrum
of computer vision and/or machine learning techniques to analyze imaging
information [19]. The methods differ mainly on the utilized features and
classifiers for obstacle representation and recognition. In respect to
performance, a direct comparison of methods is not feasible because most works
are evaluated on their own (simulated) data. In fact, there is lack or
restricted access to a common benchmark dataset with potholes and obstacles,
that can be used for comparison.

Waqa et al. [22] used superpixel segmentation to partition the image into
superpixels based on the entropy rate, and then applied Support Vector
Machines (SVM) to estimate the probability of each superpixel being the part
of some object based on textural features (namely histogram of oriented
gradients, co-occurrence matrix, intensity histogram and mean intensity). For
final object label inference, merging of the superpixels is performed using
conditional random fields to account for neighborhood similarity. The
drawbacks of this method are it’s dependency only on texture information and
more specifically the inability to distinguish between a shadow and a hole
easily, leading to potential false positives.

Another image-based method that takes advantage of the texture characteristics
of potholes is the work of Kanza et al. [23]. Here, the histogram of oriented
gradients (HOG) is extracted from the grayscale image and coupled with a Naive
Bayes classifier. If the probability calculated by the classifier is high
enough, then the pothole localization is performed using graph-based
segmentation and normalized cuts. This method presents very encouraging
results for the examined dataset that simulates a variety of cases and
conditions, including changes in illumination and potholes filled with water.

Yifan et al. [24] take a different direction and use Unmanned Aerial Vehicle
(UAV) for pothole detection in the suburb of Shihenzi City. The aerial images
are segmented and the segmented parts are used to extract features, including
the mean, standard deviation, area, length/width ratio, elliptic fit,
roundness, contrast, dissimilarity, homogeneity and correlation. Segmentation
is performed using a multiresolution segmentation algorithm that is integrated
into the eCognition Developer software (a development environment for object-
based analysis of geospatial data). For classification, the SVM, Artificial
Neural Networks (ANN), and Random Forest (RF) classifiers are compared, each
with different combinations of features while also taking the computational
time into consideration. The authors conclude that spatial features (texture
and geometry) coupled with RF are the most effective, although this method is
very sensitive to UAV image resolution, weather and lighting conditions.

Other methods focus on road cracks detection from high resolution cameras on
smartphones. Since such data are more easily available, those methods can
bypass the extraction of hand-crafted features and utilize deep architectures,
such as convolutional neural networks [25, 26, 27, 28]. However, in the case
of dense traffic situations and poor lighting conditions, techniques utilizing
images from smartphone camera are less effective.

In contrast to computer vision techniques which exploit texture information
from images, 3D point cloud processing techniques exploit the object’s
geometrical properties [15, 29, 30]. Bosurgi et al. [30] identify potholes in
road sections by estimating area, perimeter and depth information from 3D data
of pavement surfaces. Chen et al. [15] propose a framework for obstacle
detection using the pitch and rotation angles of a LiDAR sensor to create a 2D
image-like plane where the unordered set of points (from the point cloud) are
projected. From this “LiDAR-imagery” a 2D histogram is extracted and used to
find the road plane. If an adequate part of the road, in front of the vehicle
is flat, those points form a straight line in the histogram representation,
and anything above the line can be classified as a positive obstacle (points
higher than the road plane), while points below the line as a negative
obstacle (points lower than the road plane). Moreover, since water bodies
cannot be detected by LiDAR due to refraction and reflection, the authors
propose a technique to detect potholes filled with water by scanning the image
for large areas of missing data.

Shuo et al. [29] improves the aforementioned method by projecting the points
on the camera plane and interpolating the depth values of the projected points
to receive a depth image. They use both horizontal and vertical histograms to
coarsely detect the road area and refine it respectively. Although they state
their method as sensor fusion between the monocular camera and LiDAR, they do
not utilize the color values of the camera images. Both works [15] and [29]
use the KITTI dataset as a benchmark and achieve great results, comparable to
machine learning methods.

Other techniques for pothole detection may include laser scanning, ground
penetrating radar, ultrasonic sensor, as well as multi-sensor fusion,
especially concerning fusion with imaging information. An extensive review of
such techniques falls beyond the scope of this article. However, an interested
reader may be referred to the survey in [31].

####  2 Point cloud saliency

One of main challenges in techniques utilizing point clouds is the inherent
noise and the increased computational cost due to the unordered data structure
of point clouds. To address such challenges, saliency map extraction has been
proposed as a powerful step in point cloud processing to reduce noise and data
dimensionality, leading to more robust solutions and computational efficiency
[32] [33]. Yet, the use of local saliency in pothole detection has not been
sufficiently examined. Saliency maps were constructed from point clouds
obtained from Mobile Laser Scanning (MLS) in [34] for road crack detection.
MLS point clouds contain spatial information (i.e., Euclidean coordinates) and
intensity information, and thus the extracted features could leverage both
height and intensity information. Feature saliency was estimated by
calculating the distances from the normal of each point to the principal
normal of the input point clouds. In a similar setting, Wang et al. [35]
extracted saliency maps in MLS point clouds by projecting the distance of each
point’s normal vector to the point cloud’s dominant normal vector into a
hyperbolic tangent function space.

####  3 Cooperative driving

While significant advances have been made for single-agent perception, many
applications require multiple sensing agents and cross-agent communication for
more accurate results. Objects, captured by the single-agent’s sensor devices,
may be heavily occluded or far away from the sensors’ view, resulting in
sparse observations. Nevertheless, failing to detect and predict the accurate
position or moving intention of these occluded or “hard-to-see” objects might
have harmful consequences in safety-critical situations, and especially if the
reaction time is very narrow [36]. The development of multi-agent solutions
can lead to collaborative perception and, through information sharing, may
improve the driving performance and experiences, providing endless
possibilities for safe driving.

Recently, cooperative autonomous driving has been considered as a possible
solution to improve the performance and safety of autonomous vehicles [37].
Cooperative perception for 3D object detection can be performed via early or
late fusion of information, i.e., combination of multiple sensing points of
view or fusion of object detection results, respectively.Both fusion
approaches can extend the perception of the sensing system, however, only the
early fusion approach can actually exploit complementary information. A major
challenge that arises regarding cooperative perception is how to effectively
merge sensors’ data received from different vehicles to obtain a precise and
comprehensive perception outcome. Additionally, despite the attention that
cooperative driving has attracted recently, the absence of a suitable open
dataset for benchmarking algorithms has made it difficult to develop and
assess cooperative perception technologies.

Xu et al. [38] presented the first open dataset and used it to benchmark
fusion strategies for V2V (vehicle-to-vehicle) perception. They also plan to
extend the dataset with more tasks as well as sensor suites and investigate
more multimodal sensor fusion methods in the V2V and V2I (vehicle-to-
infrastructure) settings. Arnold et al. [37] proposed a system that produces a
perception of complex road segments (e.g., complex T-junctions and
roundabouts) using a network of roadside infrastructure sensors with fixed
positions. Chen et al. [39] studied the raw-data level cooperative perception
for enhancing the detection ability of self-driving systems. They fuse the
sensor data collected from different positions and angles of connected
vehicles, relying on LiDAR 3D point clouds. Liu et al. [40] addressed the
collaborative perception problem, where one agent is required to perform a
perception task and can communicate and share information with other agents on
the same task.

Chen et al. [41] proposed a point cloud feature-based cooperative perception
framework for connected autonomous vehicles to increase object detection
precision. The features are selected to be rich enough for the training
process, and at the same time have an intrinsically small size to achieve
real-time edge computing. Guo et al. [42] proposed a cooperative fusion method
to combine spatial feature maps for achieving a higher 3D object detection
performance. Yuan et al. [43] proposed a 3D keypoints feature fusion scheme
for cooperative driving detection to remedy the problem of low bounding box
localization accuracy. Fang et al. [44] presented an iterated split covariance
intersection filter-based cooperative localization strategy with a
decentralized framework. In addition, they adopted a point cloud registration
method to obtain the relative pose estimation using mutually shared
information from neighbour vehicles. Kim and Liu [45] presented the concept of
cooperative autonomous driving using mirror neuron-inspired intention
awareness and cooperative perception, providing information on the upcoming
traffic situations ahead, even beyond line-of-sight and field-of-view.

####  4 Situational awareness and AR infotainment

In the case of semi-autonomous vehicles, where the operator/driver may be
asked to take manual control of the car at any moment, it is of great
importance [46] to implement notification paradigms that direct the
operator’s, possibly reduced, attention to the event that triggered the take-
over request [47, 48]. Recently, the automotive industry started to invest
funds and efforts into AR technology and its integration with In-Vehicle
Information Systems (IVIS) for intuitive and non-intrusive information display
to the driver.

The design of AR in-vehicle systems for infotainment is a challenging task.
Rao et al. [49] performed an analysis of design methods on different use cases
aiming to identify the difficulties in implementation aspects. Despite the
vast amount of requirements for these systems to work reliably, such as
latency, bandwidth, weather conditions etc, they concluded that the
integration of augmented reality in vehicles will help drivers navigate their
environment better, and thus will be more widely adopted.

While IVIS existing in many modern vehicles with touch Liquid-Crystal Display
(LCD) displays and voice commands may seem to offer most of the utilities of
an AR infotainment system, they may actually be distracting to the driver.
David et al. [50] showed in a recent study that some IVIS require a high
cognitive demand or complex command sequences to be handled, and this can in
turn lower the awareness of the operator. This is perpetuated by the fact that
most IVIS are placed on the dashboard and usually demand their operation to
avert (even momentarily) the driver’s gaze from the road. In contrast, AR HUDs
perform information rendering on top of the environment and thus the driver
does not need to share focus in multiple locations.

The distraction potential of AR HUDs was assessed by Kim et al. [51]. An AR-
enabled windshield was used in a simulated environment with a real-life
driving video feed to test various methods of pedestrian visualization. The
gaze behavior and cognitive processes were measured and it was found that the
visual and cognitive distraction potential of AR depends on the perceptual
forms of graphical elements presented on the displays. Specifically, in some
cases visualizations, e.g., in the form of a “virtual transparent shadow”
indicating the pedestrian’s anticipated path, improved the driver’s attention
without degrading awareness of other objects or scene elements. On the other
hand, the use of bounding boxes localizing pedestrians showed to have negative
effects, because this approach either overloaded (visually) the scene or
degraded the driver’s attention on other – not highlighted but possibly
critical – scene elements. These outcomes indicate that, while the potential
of AR for improving situational awareness is tangible, a lot of attention must
be paid for the AR design to not end up cluttering and obstructing the
driver’s attention.

The research on augmented reality displays on windshields for improving driver
awareness also extends to fully Autonomous Vehicles (AV). Such informative
human-machine interfaces may help to form a mental model of the vehicle’s
sensory and planning system, thereby enhancing trust in AV, which is currently
quite low in the general public [52, 53, 54]. Lindemann et al. [55] conducted
a user study on urban environments for evaluating the situational awareness of
the driver in various scenarios. They found that their explanatory windshield
display had positive results and improved the operator’s trust. Yontem et al.
[56] also designed an AR windshield interface targeting future vehicles. Their
main focus was also to increase driver awareness by presenting graphical cues
in a non-intrusive way based on a human-centric design and taking into account
the human peripheral vision.

While the above methods provide essential feedback on the assessment of such
interfaces’ design, a significant limitation is that most studies were based
on basic or non-interactive simulations, with the steering wheel and pedals
not influencing the simulated environment and thus restricting the feeling of
immersiveness of the simulations during the evaluation study. A more
realistic, experimental study on the benefits of AR in driver’s behavior was
performed by Kim et al. [57] outdoors in a parking lot. It focused on
pedestrian collision warning based on visual depth cues delivered in a
conformal manner through a monocular display seated above the dashboard, or a
volumetric display providing binocular disparity. A limitation of this study,
which we address through our AR visualization component (subsection A of
section IV), is the limited field of view of the display used in the
experiments, potentially creating a tunneling effect of the human vision.

##  III Obstacle Detection

This section presents the proposed methodology on obstacle detection and is
followed by section IV on visualization and communication aspects. The main
components of the methodology are illustrated in the schematic diagram in Fig.
1 and can be encapsulated in the next steps:

  * •

Extraction of saliency map: A saliency value is estimated for any point of the
point cloud scene based on its local geometry, as well as the local geometry
of its neighboring points.

  * •

Scene segmentation: The estimated saliency map is then used as a feature to
segment the point cloud into areas characterizing (i) the safe area of the
road, (ii) be-aware or dangerous areas within the range of the road, and (iii)
areas out of the range of the road.

  * •

Static object recognition: Static objects (i.e., potholes and bumps) can be
identified and their point coordinates are stored and then used for the AR-
based visualization and communication to other nearby vehicles.

In this work, we assume the existence of two or more vehicles (referred as
ego1 and ego2 vehicles in this paper) that are moving on the same map of a
town but not necessarily at the same time, i.e., they are in spatial proximity
but possibly not in temporal proximity. Fig. 6 presents an example of two
registered point clouds, as received by the LiDAR devices of ego1 and ego2
vehicles, showing also their starting points (in arrows). We would like to
mention here that all the following analysis is applied to each vehicle
separately.

Figure 1: Schematic diagram of the proposed methodology.

###  A Notations

Before presenting details on the individual steps, we provide here the
necessary definitions and notations. The input data constitute a sequence of
point clouds $\mathbf{P}_{i}$, $i=1,...,l$ that represents a set of $l$
consecutive frames acquired by a LiDAR device. Each point cloud
$\mathbf{P}_{i}$ consists of $m_{i}$ vertices $\mathbf{v}$, where the value of
$m_{i}$ may be different from frame to frame. The $j$-th vertex
($\mathbf{v}_{j}$) of a point cloud $\mathbf{P}_{i}$ is represented by the
Cartesian coordinates, denoted $\mathbf{v}_{j}=\left[x_{j},\ y_{j},\
z_{j}\right]^{T},\ \forall\ j=1,\cdots,m_{i}$, where the index $i$ of the
point cloud is omitted for simplification. Thus, all the vertices can be
represented as a matrix $\mathbf{V}=\left[\mathbf{v}_{1},\ \mathbf{v}_{2},\
\cdots,\mathbf{v}_{m_{i}}\right]\in\mathbb{R}^{3\times m_{i}}$. Let’s also
denote with $\Psi_{j}^{k}$ the set of the indices of the $k$ nearest neighbors
of point $j$. For a face $f$ defined by three vertices
$(\mathbf{v}_{j1},\mathbf{v}_{j2},\mathbf{v}_{j3})$, the outward unit face
normal $\mathbf{n}_{f}$ is calculated by the following equation:

$\mathbf{n}_{f}=\frac{\left(\mathbf{v}_{j2}-\mathbf{v}_{j1}\right)\times\left(\mathbf{v}_{j3}-\mathbf{v}_{j1}\right)}{\left\|\left(\mathbf{v}_{j2}-\mathbf{v}_{j1}\right)\times\left(\mathbf{v}_{j3}-\mathbf{v}_{j1}\right)\right\|}$
(1)

The point normal $\mathbf{n}_{j}$, representing the normal of each point
separately, is calculated as:

$\mathbf{n}_{j}=\frac{\sum_{\forall\mathbf{n}_{f}\in\Psi_{j}^{k}}\mathbf{n}_{f}}{|\Psi_{j}^{k}|}$
(2)

###  B Saliency Map Estimation of the Point Cloud Scene

The purpose of this step is to calculate a metric of saliency for each vertex
of a point cloud. Assuming point clouds without context information, saliency
characterizes the geometric properties in a local neighborhood of points,
i.e., high saliency values represent more perceptually prominent vertices
which usually correspond to sharp corners (high-frequency spatial
information). On the opposite, the geometrically least important points are
those that lie in flat areas.

For the estimation of the saliency map, we implemented and modified the fusion
technique presented in [33]. Instead of using guided normals of centroids, as
in the original version [33], we now utilize normals for the points. This was
performed to accelerate computations. Since the number of faces is usually
approximately twice the number of vertices, the point normals are almost half
the number of the centroid normals. For the sake of completeness, we present
here our approach for the estimation of the saliency map of a point cloud
scene, utilizing point normals.

Our fusion technique combines geometric saliency ($s^{(1)}$) with spectral
saliency ($s^{(2)}$) features. The unique characteristics of each of these
saliency features make the methodology more robust to point clouds acquired
under real conditions, thereby being potentially affected by noise and
outliers. The method processes each frame independently without examining past
temporal information. Thus, as the methodology is applied for each point cloud
in the sequence independently, for simplicity we omit the index $i$
(indicating the frame number) from now on in the equations.

For a point cloud $\mathbf{P}$ with $m$ vertices, a matrix $\mathbf{E}\in\
\mathbb{R}^{3m\times(k+1)}$ is constructed which includes in the first column
the $m$ point normals
($\mathbf{n}_{j}=[{n}_{j_{x}},{n}_{j_{y}},{n}_{j_{z}}]^{T}$) of each vertex
$j$, $j=1,\cdots,m$, respectively, and in the subsequent $k$ columns the point
normals of the $k$ nearest neighbors of vertex $j$ (i.e.
$\mathbf{n}_{j\kappa}\in\Psi_{j}^{k}$). The salient features extracted by this
approach capture global information since the matrix $\mathbf{E}$ is
constructed using the point normals of the whole scene.

In order to exploit the geometrical coherence between neighboring normals, we
apply Robust Principal Component Analysis (RPCA) to decompose the matrix
$\mathbf{E}$ into a low-rank matrix $\mathbf{L}\in\
\mathbb{R}^{3m\times(k+1)}$ and a sparse matrix $\mathbf{S}\in\
\mathbb{R}^{3m\times(k+1)}$, as described in the appendix A. The matrix
$\mathbf{L}$ consists of the low-rank values $\bar{\mathbf{n}}$ of the point
normals $\mathbf{n}$, while the matrix $\mathbf{S}$ consists of the
corresponding sparse values represented as $\dot{\mathbf{n}}$. The values of
this matrix are zero (or to be more specific nearly zero) if the row
(representing a neighboring patch of points) corresponds to point normals with
very similar values, i.e., the vertex lies in a flat area, and very large
values if the row corresponds to point normals with big dissimilarity (i.e.,
the vertex lies in a very sharp corner). The fact that most of the local
patches $\Psi_{j}^{k}$ of a 3D surface are piecewise flat confirms that the
matrix $\mathbf{S}$ can be considered a sparse matrix.

$\mathbf{S}=\begin{bmatrix}\dot{\mathbf{n}}_{1}&\dot{\mathbf{n}}_{{11}}&\dot{\mathbf{n}}_{{12}}&\dots&\dot{\mathbf{n}}_{{1k}}\\\
\dot{\mathbf{n}}_{2}&\dot{\mathbf{n}}_{{21}}&\dot{\mathbf{n}}_{{22}}&\dots&\dot{\mathbf{n}}_{{2k}}\\\
\vdots&\vdots&\vdots&\ddots&\vdots\\\
\dot{\mathbf{n}}_{{m}}&\dot{\mathbf{n}}_{{m1}}&\dot{\mathbf{n}}_{{m2}}&\dots&\dot{\mathbf{n}}_{{mk}}\\\
\end{bmatrix}$ (3)

In other words, sparsity of the matrix is assumed because piecewise flat areas
are the most dominant geometrical pattern in a 3D surface.

####  B1 Estimation of the geometrical saliency (global approach)

As the similarity of normals between neighboring points is a measure of
geometrical coherence of the local neighborhood, we estimate the sparsity of
the dissimilarity of normals and use it as a feature for geometrical saliency,
$s^{(1)}$.Low values of the sparse matrix indicate that the normals of the
point and its neighbors are similar (low-rank). This means that if all points
in a neighborhood have similar geometrical characteristics, the respective
patch represents a flat area. On the opposite, high dissimilarity indicates
that the surface has an irregular shape. For a point $\mathbf{v}_{j}$ the
geometric saliency feature, $s_{j}^{(1)}$, is estimated by the values of the
first column of the sparse matrix $\mathbf{S}$ according to:

$s_{j}^{(1)}=||\dot{\mathbf{n}}_{j}||^{2}=\sqrt{\dot{n}_{j_{x}}^{2}+\dot{n}_{j_{y}}^{2}+\dot{n}_{j_{z}}^{2}}\
\ \forall\ j=1,\cdots,m$ (4)

where $\dot{n}_{j_{x}}$ denotes the scalar value of the $x$ coordinate, of the
$[3\cdot(j-1)+1]^{th}$ row, of the $1^{st}$ column of the $\mathbf{S}$ matrix.

####  B2 Estimation of the spectral saliency (local approach)

For the estimation of the spectral-based saliency, $s_{j}^{(2)}$, for a vertex
$j$ of the point cloud, we use the submatrix $\mathbf{E}_{j}\ \in\
\mathbb{R}^{3\times(k+1)}$, that includes the 3 corresponding rows of the
matrix $\mathbf{E}$:

$\mathbf{E}_{j}=\begin{bmatrix}{n}_{j_{x}}&{n}_{j_{x1}}&{n}_{j_{x2}}&\dots&{n}_{j_{xk}}\\\
{n}_{j_{y}}&{n}_{j_{y1}}&{n}_{j_{y2}}&\dots&{n}_{j_{yk}}\\\
{n}_{{j_{z}}}&{n}_{j_{z1}}&{n}_{j_{z2}}&\dots&{n}_{j_{zk}}\\\
\end{bmatrix},\begin{array}[]{c}\forall\ j=1,\cdots,m\end{array}$ (5)

In other words, each submatrix $\mathbf{E}_{j}$, which is a subset of the
global matrix $\mathbf{E}_{i}$, consists of the point normals of a local
neighborhood of the vertex $\mathbf{v}_{j}$. Then, for each one of these local
matrices $\mathbf{E}_{j}$, the covariance matrix
$\mathbf{R}_{j}\in\mathbb{R}^{3\times 3}$ is calculated:

$\mathbf{R}_{j}=\mathbf{E}_{j}\mathbf{E}_{j}^{T}$ (6)

Next, the calculated matrix $\mathbf{R}_{j}$ is decomposed into a matrix
$\mathbf{U}$ consisting of the eigenvectors and a diagonal matrix
$\mathbf{\Lambda}=\text{diag}(\lambda_{j1},\lambda_{j2},\lambda_{j3})$
consisting of the corresponding eigenvalues, i.e., $[\mathbf{U}\ \
\mathbf{\Lambda}]=\text{eig}(\mathbf{R}_{j})$, where eig(.) represents the
eigendecomposition operation.

Finally, the spectral saliency of each vertex is calculated by the inverse
$l^{2}$-norm of the corresponding eigenvalues:

$s_{j}^{(2)}=\frac{1}{\sqrt{\lambda_{j1}^{2}+\lambda_{j2}^{2}+\lambda_{j3}^{2}}}\
\forall\ j=1,\cdots,m$ (7)

Eq. (7) indicates that large values of the term
$\sqrt{\lambda_{i1}^{2}+\lambda_{i2}^{2}+\lambda_{i3}^{2}}$ correspond to
small saliency features implying that the centroid lies in a flat area, while
small values of the eigenvalues’ norm correspond to large saliency,
characterizing the specific centroid as a discriminative point.

This can be easily justified by the fact that a point normal lying on a flat
area is represented by one dominant eigenvector, the corresponding eigenvalue
of which has a very large value (especially, considering that it is squared).
On the other hand, the point normal of a vertex lying on a corner is
represented by three eigenvectors, that correspond to eigenvalues with small
and almost equal amplitude, as shown in Fig. 2.

Figure 2: (a) Cube model, (b) corner
($\lambda_{i1}\cong\lambda_{i2}\cong\lambda_{i3}$), (c) edge
($\lambda_{i1}\cong\lambda_{i2}>\lambda_{i3}$), (d) flat area
($\lambda_{i1}>\lambda_{i2}\cong\lambda_{i3}$).

####  B3 Normalization and fusion of local and global saliency

Finally, we linearly scale the values of the geometric ($s^{(1)}$) and
spectral ($s^{(2)}$) saliency in the range of [0-1] and combine them through
weighted averaging according to:

$s_{j}=\frac{w_{1}\bar{s_{j}}^{(1)}+w_{2}\bar{s_{j}}^{(2)}}{w_{1}+w_{2}}\ \
\forall\ j=1,\cdots,m_{i}$ (8)

where $\bar{s}^{(1)}$ and $\bar{s}^{(2)}$ denote the normalized geometric and
spectral saliency features, and $w_{1}$ and $w_{2}$ the corresponding weights.
We note here that we used equal weights ($w_{1}=w_{2}=1$) in all of our
experiments, however, the weights can be tuned to emphasize the local or
global saliency descriptors, respectively.

The proposed method has shown to be robust [33, 32], even for complex surfaces
with different geometrical characteristics and patterns, since it exploits
spectral properties (i.e., sensitivity in the variation of neighboring
normals) and geometrical characteristics (i.e., sparsity of intense prominent
spatial features). An example of the visualization of the saliency map, as
applied to the point cloud of a scene shown in Fig. 3), is presented in Fig.
4.

Figure 3: Image from the camera of the vehicle, the texture of a pothole is
also apparent. Figure 4: Example of saliency map extracted from the road scene
shown in Fig. 3.

###  C Scene Segmentation for the Identification of On-road Obstacles

The saliency map of each frame is used to categorize different regions of the
scene. For illustration purposes the regions are visualized in different
colors:

  * •

Blue: The safe area of the road beyond the view of the driver.

  * •

Yellow: Be-aware areas representing negative obstacles.

  * •

Cyan: Hazardous areas in the range of the road representing positive
obstacles.

  * •

Purple: Dangerous areas outside of the range of the road.

  * •

Red: Recognized obstacles in the range of the road (e.g., potholes).

To define the vehicle’s moving direction steering data are used received by
internal sensors of the vehicle. The direction of the vehicle specifies which
part of the scene in the field of view is in front of the vehicle and is used
as as parameter, in addition to saliency mapping, for the segmentation of the
point cloud. The more critical regions are the ones that lie within the limits
of the road. A segmentation example is illustrated in Fig. 5.

Figure 5: Segmentation of the point cloud scene based on the saliency map in
Fig. 4 and the vehicle’s moving direction. Figure 6: Point cloud map of both
two vehicles (ego1 and ego2). Figure 7: Example of segmentation of the point
cloud projected to the AR interface (in the view of ego1). Figure 8:
Perspective projection of the point cloud vertices to the AR interface and
image filling (in the view of ego1). Figure 9: Pothole recognition
(highlighted in red color) and AR visualization of the corresponding
information (in the view of ego1). Figure 10: AR projection of the point cloud
vertices to the scene image that depicts the starting point of view of the
ego2 vehicle. Figure 11: Early warning of upcoming pothole to inform ego2.
Figure 12: Pothole recognition and visualization (in the view of ego2).

###  D Data simulations

For evaluation of our methodology, we created a rich dataset using CARLA, an
open-source autonomous driving simulator [16]. CARLA is based on a server-
client system, in which the server is responsible for running the simulations
including the calculation of physics, weather conditions, collision detection
and sensor readings. It operates on the OpenDRIVE specification [58] for
defining junctions, traffic lights, etc, and is used by CARLA for simulating
independent agents, such as other cars and pedestrians. This makes CARLA ideal
for creating complex scenarios and realistic driving conditions for our tests.

The server running the simulations is powered by Unreal Engine. Clients can
connect and request changes to almost any element in the world being essential
for the creation of scenarios. They also receive sensor data and manage input
to the vehicle controlled by the user. CARLA supports a wide range of sensor
suites with extensive configurability to its intrinsic parameters. In our
work, we use a LiDAR sensor on top of the vehicle and a monocular RGB camera,
placed in the front part of the car, for simulated data collection. By placing
these sensors in an autonomous car and initiating its navigation in the
virtual environment, we were able to create a very large dataset for
evaluating our algorithms. In the future, we plan to assess the AR
visualization effectiveness, with respect to reaction time and awareness
increase, in a real environment with a driver manually controlling a vehicle.

#### Contributions in CARLA simulator

Due to lack of benchmark point clouds datasets representing real road scenes
with obstacles (potholes and bumps), we used the CARLA simulator to create
obstacle-free environment data, in which we subsequently introduced simulated
obstacles. Specifically, we designed obstacles as curved point cloud surfaces
using the open-source software Blender555https://www.blender.org/ and used
them to substitute parts of the road. To avoid modeling the obstacles by hand,
we followed an automated procedure to generate a plethora of different
obstacles based on several parameters, such as depth, ellipticity and size. An
example of a frame in the CARLA simulator with a simulated pothole is
presented in Fig. 3 (texture) and in Fig. 4 (geometry).

##  IV Interfaces and Communication

Context-awareness is a critical factor for successful take-over requests and a
lot of effort has been devoted to determining the type of stimulus (e.g.
visual, auditory, vibrotactile) [59] and the required time-window [60, 61,
62]. In the case of partial or conditional driving automation, our framework
could be used to prepare the driver to quickly take the control of the
vehicle, if requested. In order to ensure that the driver is able to swiftly
take over the control of the vehicle in an efficient way, we developed a
notification system that presents relevant information about the condition of
the environment. Our notification system is based on non-intrusive visual cues
to prevent tunnel visioning, alerting the driver of potential risks and also
directing his/her attention to the objects of interest that sparked the take-
over request. In that way, in addition to assisting the human operator during
manual driving, the system can, in times of automated driving, trigger the
attention of the operator to possible external hazards and preparing him/her
to resume control.The visualization technique presented in this section is
designed as an AR windshield interface, although this is not restrictive, i.e.
the method can be implemented in any AR interface.

###  A AR Visualization

The visualization of obstacles is performed by projection. Assuming the
position is known for the AR interface and the LiDAR relative to the world, we
construct a transformation matrix to map the points of the point cloud from
the LiDAR relative coordinate system to the AR interface’s coordinate system.
The transformation between two different coordinate systems is typically
performed by applying serially a scale, a rotation and then a translation
transformation. Since both coordinate systems are orthonormal, the scaling can
be omitted. Also, by taking advantage of the rigid body nature of the vehicle
where the LiDAR and AR interface is located, we also omit the rotation matrix
given that, without loss of generality, we can assume that the two coordinate
systems are aligned. According to these assumptions, the LiDAR coordinates are
transformed into the AR interface’s coordinates by a simple translation.

For projecting the points of the point cloud to the AR interface, we assume a
simple pinhole camera model. If the AR interface is, for example, an AR
windshield, then the windshield represents the image plane and the head of the
driver the principal point with coordinates $(x_{0},y_{0})$. That way, the
focal distance $f=(f_{x},f_{y})$ represents the distance from the driver to
the image plane. With the dimensions of the image plane (windshield), and
specifically the aspect ratio, known, the frustum is fully defined and the
projection can be made from a point in 3D windshield coordinates $(x,y,z)$ to
pixels $(u,v)$ on the image plane using the following equation:

$\centering\begin{pmatrix}u\\\ v\end{pmatrix}=\begin{pmatrix}1&0&x_{0}\\\
0&1&y\\\ 0&0&0\end{pmatrix}\begin{pmatrix}f_{x}&0&0\\\ 0&f_{y}&0\\\
0&0&0\end{pmatrix}\begin{pmatrix}x\\\ y\\\ z\end{pmatrix}\@add@centering$

An undesirable property is the sparsity of the projected pixels attributed to
the sparsity of the point cloud. To overcome this limitation, we use an
iterative nearest neighbour algorithm on the image space to fill the gaps
between projected points. The result of this process is shown in Figs. 7-12.

More specifically, Fig. 7 and Fig. 10 illustrate the segmented point cloud
projected to the AR interface of ego1 and ego2 correspondingly. Note that all
information is rendered for the sake of completeness. In real-world cases only
the necessary information (e.g., arrows or recognised potholes) will be
rendered so as to avoid clutter. Fig. 8 shows the perspective projection of
the points to the AR interface and image filling for ego1. In Fig. 9 and Fig.
12, the pothole recognition and visualization is depicted for the vehicles
ego1 and ego2, while a warning about an upcoming pothole (retrieved from the
database) before reaching the field of view of ego2 is presented in Fig. 11.

We would like to clarify here that for evaluation of our methodology and
demonstration purposes in the previous figures we project and illustrate in
the 2D display device all the information from scene segmentation. However, in
real driving scenarios only the most relevant information of the scene (e.g.,
dangerous objects, potholes) would be highlighted and displayed so as to
decrease the amount of any unnecessary information that may bother or confuse
the driver.

###  B Information Storage and Vehicle Communication Rules

One of the advantages of autonomous vehicles is their ability to communicate
with each other forming a cyber-physical system of systems. Many new
opportunities arise from the ability of systems to share information, one of
which is the transmission of objects or landmarks of interest that were
previously observed by an agent, to other agents of the system who could
benefit from such information. In particular, our work focuses on information
sharing among vehicles about encountered obstacles, such as potholes and
bumps, through a centralized server. When a vehicle identifies an unexpected
(i.e., unregistered) obstacle, the vehicle sends a request to the server and
after further inspection, the new potential obstacle is either discarded or
added to the database. Vehicles may also send information regarding already
known obstacles when they come across them. Such information includes the
Global Positioning System (GPS) location, dimensions and geometrical
characteristics in case the obstacle needs updating in the database, e.g. it
has increased in size or has been fixed. Through this communication system, a
driver can be warned about potential hazards that may not yet be in his field
of view or they are obstructed by other objects and thus, increase his
performance and decision-making abilities. We should clarify that our work
does not focus on communication protocols and defence mechanisms against
potential network attacks, but rather defines a solid framework describing the
roles of each node and the information flow.

By using the LiDAR-based obstacle detection method, described in section III,
the vehicle transmits via a communication component to a central server the
points belonging to the obstacle, segmented from the point cloud scene. The
information is coupled with a timestamp and the GPS location of the vehicle at
that instance. The server then transmits to any vehicle in the vicinity of the
obstacle, alerting (autonomous vehicles or human operators) about potential
hazards from a large distance and thus helping alleviate the inability of the
LiDAR sensor to identify obstacles from such a range. In the case of a driver,
we also use the AR interface of the vehicle to display, in a non-distracting
manner, the location and nature of the potentially upcoming obstacle.

Potholes can change shape over time, most commonly due to deterioration of the
surrounding pavement and erosion caused by environmental effects or in the
opposite case due to pothole repair. Thus, periodic updates are necessary for
the long-term reliability of the pothole visualization component. As there is
a need for periodical evaluation of the objects in the server database and
update in the case of changes, we assign a shape- and geometry-based
descriptor at each obstacle, so that it is characterized by a unique
representative signature. Thus, every vehicle encountering the obstacle in a
nearby range, calculates the descriptor of the obstacle’s area. The new
descriptor is then transmitted to the server and is used to confirm whether
the information is up-to-date. In the case of a difference in the descriptor’s
value, an algorithm running in the server decides between keeping the old
descriptor, updating it with the new one, or marking the obstacle as removed
and deleting the entry from the database.

More specifically, we implement a simple system that (when a new pothole is
detected) initiates a database search to retrieve whether the pothole is new
or already existed and needs to be updated. Since potholes are static and thus
change only in shape, the similarity check is based only on the bounding box
of the re-identified pothole. When the overlap of the bounding boxes is less
than a threshold, the previous object is replaced by the new one. In our
experiments we used a threshold of 15% reshape in the area in either direction
to avoid frequent unnecessary updates, while also retaining the required
precision in representation. Similarly, the algorithm checks for significant
changes in the bounding box dimensions. A flowchart showcasing the information
update and communication pipeline between two vehicles is shown in Fig. 13.

Figure 13: Flowchart of communicative vehicles for obstacle sharing. Figure
14: Pothole detection in point cloud data of real potholes [63]. Two dense
models are visualized: model1 (rows 1-3) and model2 (rows 4-6). For each
model, the three rows illustrate (i) the ground truth, (ii) the heatmap
visualizing the saliency map of the pothole and (iii) the estimated point
cloud, respectively. The columns show results with decreasing density
resolutions (in respect to the original model): (a) original model, (b) $\sim
50\%$ of the vertices, (c) $\sim 10\%$ of the vertices, (d) $\sim 5\%$ of the
vertices.

##  V Experimental Analysis

In this section, we will present and discuss in detail the experimental
analysis and will evaluate our proposed framework.

###  A Experimental Setup, Datasets and Metrics

The experiments were carried out on an Intel Core i7-4790HQ CPU @ 3.60GHz PC
with 16 GB of RAM. The core algorithms are written in Matlab and C++. The
evaluation of the methodology was performed using (i) synthetic dataset of
potholes that we have created and (ii) 3D point cloud potholes from real
datasets with known models (used as ground truth) which have been evaluated by
other methods too [63, 64, 65, 66].

The pothole detection algorithms are compared in terms of the pixel-level (for
image-based methods) and point-level (for point clouds)
$precision=[TP/(TP+FP)]$, $recall=[TP/(TP+FN)]$,
$accuracy=[(TP+TN)/(TP+TN+FP+FN)]$ and $F-score=2\cdot[(precision\cdot
recall)/(precision+recall)]$, where $TP,\ FP,\ TN,\ FN$, represent the number
of True-Positive, False-Positive, True-Negative and False-Negative pixels,
respectively. The positive class includes all vertices belonging to the
pothole ($P$) and the negative class all vertices belonging to the road ($R$).
The performance metrics can also be expressed as shown in Table I, where Real
Pothole (RP) represents the recall or in other words the percentage of
vertices correctly annotated as pothole, Real Road (RR) represents the
percentage of vertices correctly annotated as road, Not real Pothole (NP)
represents the percentage of vertices wrongly annotated as pothole and Not
real Road (NR) represents the percentage of vertices wrongly annotated as
road.

TABLE I:  Evaluation metrics for pothole detection (in percentage %)

($\times 100\%$) | Annotated as Pothole | Annotated as Road
---|---|---
Actual Pothole | $\text{RP}=\frac{\text{TP}}{\text{P}}$ | NR = 1 - RP
Actual Road | NP = 1 - RR | $\text{RR}=\frac{\text{TN}}{\text{R}}$

TABLE II:  Pothole detection accuracy (in percentage %) for different density resolutions of the point cloud models. Models | Original |  $\sim 0.5\ *$ Original |  $\sim 0.1\ *$ Original |  $\sim 0.05\ *$ Original
---|---|---|---|---
Model 1 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.44 | RR = 99.56 |  NP = 0.40 | RR = 99.60 |  NP = 0.68 | RR = 99.32 |  NP = 0.89 | RR = 99.11
Model 2 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 99.38 | NR = 0.62 |  RP = 100 | NR = 0
NP = 0.65 | RR = 99.35 |  NP = 0.58 | RR = 99.42 |  NP = 0.98 | RR = 99.02 |  NP = 1.22 | RR = 98.78
Model 3 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.41 | RR = 99.59 |  NP = 0.36 | RR = 99.64 |  NP = 0.63 | RR = 99.37 |  NP = 0.74 | RR = 99.26
Model 4 |  RP = 99.91 | NR = 0.08 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.45 | RR = 99.55 |  NP = 0.41 | RR = 99.59 |  NP = 0.76 | RR = 99.24 |  NP = 0.93 | RR = 99.07
Model 5 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 99.36 | NR = 0.64 |  RP = 100 | NR = 0
NP = 0.62 | RR = 99.38 |  NP = 0.56 | RR = 99.44 |  NP = 0.86 | RR = 99.14 |  NP = 1.25 | RR = 98.75
Model 6 |  RP = 100 | NR = 0 |  RP = 99.57 | NR = 0.43 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.52 | RR = 99.48 |  NP = 0.46 | RR = 99.54 |  NP = 0.78 | RR = 99.22 |  NP = 1.22 | RR = 98.78
Model 7 |  RP = 99.92 | NR = 0.08 |  RP = 100 | NR = 0 |  RP = 99.15 | NR = 0.85 |  RP = 100 | NR = 0
NP = 0.47 | RR = 99.53 |  NP = 0.44 | RR = 99.56 |  NP = 0.80 | RR = 99.20 |  NP = 1.10 | RR = 98.90
Model 8 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.48 | RR = 99.52 |  NP = 0.44 | RR = 99.56 |  NP = 0.73 | RR = 99.27 |  NP = 1.06 | RR = 98.94
Model 9 |  RP = 100 | NR = 0 |  RP = 99.13 | NR = 0.87 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.45 | RR = 99.55 |  NP = 0.37 | RR = 99.63 |  NP = 0.65 | RR = 99.35 |  NP = 0.96 | RR = 99.04
Model 10 |  RP = 99.92 | NR = 0.08 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.42 | RR = 99.58 |  NP = 0.38 | RR = 99.62 |  NP = 0.68 | RR = 99.32 |  NP = 0.84 | RR = 99.16
Model 11 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 98.56 | NR = 1.44
NP = 0.59 | RR =99.41 |  NP = 0.51 | RR = 99.49 |  NP = 0.95 | RR = 99.05 |  NP = 1.03 | RR = 98.97
Model 12 |  RP = 100 | NR = 0 |  RP = 99.36 | NR = 0.64 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.44 | RR = 99.56 |  NP =0.36 | RR = 99.64 |  NP = 0.64 | RR = 99.36 |  NP = 0.81 | RR = 99.19
Model 13 |  RP = 99.96 | NR = 0.04 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0 |  RP = 100 | NR = 0
NP = 0.42 | RR = 99.58 |  NP = 0.38 | RR = 99.62 |  NP = 0.65 | RR = 99.35 |  NP = 0.79 | RR = 99.21
Average |  RP = 99.98 | NR = 0.02 |  RP = 99.85 | NR = 0.15 |  RP = 99.83 | NR = 0.17 |  RP = 99.88 | NR = 0.11
NP = 0.49 | RR = 99.51 |  NP = 0.43 | RR = 99.57 |  NP = 0.75 | RR = 99.25 |  NP = 0.99 | RR = 99.01%
Figure 15: Pothole detection on real data [64]. (a) RGB images of potholes,
(b) corresponding point cloud of potholes with texture, (c) point cloud of
potholes, (d) ground truth binary mask of potholes, (e) estimated binary mask
of potholes, (f) enlarged details of the ground truth point cloud, (g)
enlarged details of the estimated point cloud. TABLE III: Comparison of the
pothole detection accuracy among different state-of-the-art approaches.

Dataset | Method | Correct Detection | Incorrect | Misdetection | Recall | Precision | Accuracy | F-score
---|---|---|---|---|---|---|---|---
1 | 1 [67] | 11 | 11 | 0 | 0.520 | 0.543 | 0.989 | 0.531
2 [68] | 22 | 0 | 0 | 0.462 | 0.998 | 0.994 | 0.632
3 [64] | 22 | 0 | 0 | 0.499 | 0.987 | 0.994 | 0.663
4 [63] | 21 | 1 | 0 | 0.701 | 0.964 | 0.995 | 0.811
our | 22 | 0 | 0 | 0.853 | 0.993 | 0.991 | 0.918
2 | 1 [67] | 42 | 10 | 0 | 0.975 | 0.971 | 0.999 | 0.973
2 [68] | 40 | 8 | 4 | 0.874 | 0.991 | 0.997 | 0.929
3 [64] | 51 | 1 | 0 | 0.980 | 0.980 | 0.999 | 0.980
4 [63] | 52 | 0 | 0 | 0.950 | 0.883 | 0.992 | 0.915
our | $40^{2}$ | 0 | 0 | 0.909 | 0.996 | 0.992 | 0.951
3 | 1 [67] | 5 | 0 | 0 | 0.612 | 0.771 | 0.995 | 0.683
2 [68] | 5 | 0 | 0 | 0.534 | 0.992 | 0.996 | 0.694
3 [64] | 5 | 0 | 0 | 0.582 | 0.983 | 0.996 | 0.731
4 [63] | 5 | 0 | 0 | 0.702 | 0.996 | 0.996 | 0.823
our | 5 | 0 | 0 | 0.953 | 0.984 | 0.996 | 0.969
Total | 1 [67] | 58 | 21 | 0 | 0.800 | 0.822 | 0.994 | 0.800
2 [68] | 67 | 8 | 4 | 0.695 | 0.992 | 0.995 | 0.817
3 [64] | 78 | 1 | 0 | 0.771 | 0.982 | 0.996 | 0.864
4 [63] | 78 | 1 | 0 | 0.890 | 0.898 | 0.996 | 0.894
our | 67 | 0 | 0 | 0.899 | 0.994 | 0.992 | 0.945

2 Only 40 of the refereed (52) models were founded online.

###  B Results

For the evaluation of our method, two public available datasets [63, 64] were
utilized providing point clouds of real potholes. Fig. 14 visualizes results
of our pothole detection method for the dataset created by real potholes [63]
under different density resolutions (14 (a)-(d)). Points in red represent the
vertices belonging to the pothole, while points in blue represent vertices
belonging to the road, both for the ground truth and the estimated point
clouds. Two dense models (14 (a)) are utilized as presented in rows 1-3 and
4-6, respectively. To investigate the performance of our approach in more
realistic conditions, we increasingly downsampled the original point cloud (14
(b)-(c)) to evaluate the robustness of detection of our algorithm. The
corresponding number of vertices for the two models (original and downsampled)
are shown above each model, respectively. The heatmap (rows 2 and 5)
illustrates the geometric and spectral saliency per vertex (as estimated from
Eq. 8). Higher salient values are depicted with deep red color while lower
salient values with deep blue.

Due to the sensitive nature of the specific application involving safety of
drivers (via information visualization for situational awareness), we prefer
our algorithm to provide a small percentage of NR than having even a small
value of NP (please refer to Table I). To wrongly identify as a pothole a
small area of the road around an actual pothole is not as critical in our
application as the opposite, namely to fail to present or partially present a
potentially dangerous object (e.g., pothole, ramp).

The detailed results with all evaluation metrics are shown in Table II for
each of the thirteen 3D models of the point cloud dataset, and under different
point cloud density resolutions. The results of this table show that our
method is robust even for very low point cloud density. This is an important
observation, since the output of the LiDAR device has a low density resolution
pattern.

Fig. 15 visualizes some examples of the pothole detection algorithm applied in
an other dataset [64]. The first column of this figure illustrates the RGB
image presenting real road potholes. In the second column (Fig. 15-(b)), the
corresponding point cloud with the relative texture is presented. The geometry
represented by the 3D coordinates of the point cloud (without any color
information) is presented in Fig. 15-(c). Fig. 15-(d) shows the ground truth
vertices (in red) representing the pothole and Fig. 15-(e) presents our
pothole estimation result. Figs. 15-(f) $\&$ (g) just present enlarged details
of Figs. 15-(d) $\&$ (e), respectively, for easier visual comparison.

Table III provides a qualitatively comparison of our method versus other
approaches of the literature. However, it should be mentioned that the results
are not directly comparable because the other methods use only the visual
information of the RGB images, while our method uses only the geometrical
information of the corresponding point cloud.

##  VI Conclusions

In this paper we presented a methodology for identification of road obstacles
and their AR-based visualization targeting both the driver of the ego
vehicleand other drivers in a spatial vicinity whose LiDAR device has not
captured the obstacle information yet. AR-enabled technologies (beyond current
AR headsets) are expected to be utilized in the near future for providing
guidance to the drivers [69, 70, 71], increasing their situational awareness,
and facilitating cooperation with other vehicles and road users (e.g.,
pedestrians, bicycles). The main purpose of the proposed system is to be
capable to provide in real-time information to the drivers of autonomous and
connected vehicles in cooperative driving situations, in order to increase
their situational awareness.

Main emphasis was placed to the detection of potholes rather than protruding
obstacles, because missing parts of the road present particular challenges
that have not been handled efficiently by the available methods so far [8].
Our method is based on the analysis of point clouds which is challenged by the
lack of benchmark datasets obtained from LiDAR devices. To overcome this
problem, we created our own synthetic dataset and added it to the maps of the
CARLA simulator, thereby creating realistic driving environments. The
comparison of our method with other state-of-the-art approaches, regarding the
accuracy of pothole detection in real datasets, has shown its effectiveness
providing very promising outcomes.

Our future plans include the visualization of additional information that can
facilitate the increase of driver’s situational awareness (e.g., road
boundaries), and the analysis of user preferences, e.g., via questionnaires,
of the AR visualization system when driving (through a steering wheel chair)
in the simulated environment of the CARLA simulator.

## Appendix

###  A Robust Principal Component Analysis (RPCA)

RPCA is a powerful mathematical tool that has been used in many scientific
domains in order to decompose an observed measurement $\mathbf{E}$ into a low-
rank matrix $\mathbf{L}$, representing the ideal data unaffected by any kind
of noise, and a sparse matrix $\mathbf{S}$, representing the noisy data.
Decomposition is performed by solving the following equation:

$\operatorname*{arg\,min}_{\mathbf{L},\mathbf{S}}\|\mathbf{L}\|_{*}+\lambda\|\mathbf{S}\|_{1},\
\ \ \text{s.t.}\ \mathbf{L}+\mathbf{S}=\mathbf{E},$ (9)

where $\|\mathbf{L}\|_{*}$ is the nuclear norm of a matrix $\mathbf{L}$ (i.e,
$\sum_{i}\sigma_{i}(\mathbf{L})$ is the sum of the singular values of
$\mathbf{L}$).

A lot of works have been proposed all of these years, presenting excellent
results. However, despite the effectiveness that some works [72], [73] have
presented in the past, the execution times of the proposed algorithms need
improvement. This convex problem can be solved using a very fast approach, as
described in [74], according to:

$\operatorname*{arg\,min}_{\mathbf{L},\mathbf{S}}\frac{1}{2}\|\mathbf{L}+\mathbf{S}-\mathbf{E}\|_{F}+\lambda\|\mathbf{S}\|_{1}\
\ \ \text{s.t.}\ \text{rank}(\mathbf{L})=K$ (10)
$\mathbf{L}^{(t+1)}=\operatorname*{arg\,min}_{\mathbf{L}}\|\mathbf{L}+\mathbf{S}^{(t)}-\mathbf{E}\|_{F}\
\ \ \text{s.t.}\ \text{rank}(\mathbf{L})=K$ (11)
$\mathbf{S}^{(t+1)}=\operatorname*{arg\,min}_{\mathbf{S}}\|\mathbf{L}^{(t+1)}+\mathbf{S}-\mathbf{E}\|_{F}+\lambda\|\mathbf{S}\|_{1}$
(12)

In each $(t)$ iteration, the Eq. (11) is updated with $\text{rank}=K$. If
$\frac{u_{K}}{\sum_{i=1}^{K}u_{i}}>\epsilon$, where $u$ denotes the singular
values and $\epsilon$ is a small threshold, then the rank is increased by one
(i.e., $K=K+1$) and the Eq. (12) is updated too. To update the Eq. (11), a
partial $\text{SVD}(\mathbf{E}-\mathbf{S}^{(t)})$ is estimated keeping $K$
components. To update the Eq. (12), a shrinkage operator is used
$\mathcal{D}(.)$, where:

$\mathcal{D}(\mathbf{E}-\mathbf{L}^{(t+1)},\lambda)=\text{sign}(\mathbf{E}-\mathbf{L}^{(t+1)})\text{max}\\{0,|\mathbf{E}-\mathbf{L}^{(t+1)}|-\lambda\\}$
(13)

## References

  * [1] “Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles,” _SAE Standard J3016201806_ , 2018.
  * [2] J. Leng, Y. Liu, D. Du, T. Zhang, and P. Quan, “Robust obstacle detection and recognition for driver assistance systems,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 4, pp. 1560–1571, 2020.
  * [3] T. L. Overton, T. E. Rives, C. Hecht, S. Shafi, and R. R. Gandhi, “Distracted driving: prevalence, problems, and prevention,” _International Journal of Injury Control and Safety Promotion_ , vol. 22, no. 3, pp. 187–192, 2015, pMID: 24499372. [Online]. Available: https://doi.org/10.1080/17457300.2013.879482
  * [4] S. McEvoy and M. Stevenson, “An exploration of the role of driver distraction in serious road crashes,” in _International Conference on the distractions in driving, Sydney, New South Wales, Australia_ , 2007, pp. 189–211.
  * [5] I. Dua, A. U. Nambi, C. V. Jawahar, and V. Padmanabhan, “Autorate: How attentive is the driver?” in _2019 14th IEEE International Conference on Automatic Face Gesture Recognition (FG 2019)_ , 2019, pp. 1–8.
  * [6] I. Radu and B. Schneider, “How augmented reality (ar) can help and hinder collaborative learning: A study of ar in electromagnetism education,” _IEEE Transactions on Visualization and Computer Graphics_ , pp. 1–1, 2022\.
  * [7] A. Dhiman and R. Klette, “Pothole detection using computer vision and learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 8, pp. 3536–3550, 2020.
  * [8] T. Sun, W. Pan, Y. Wang, and Y. Liu, “Region of interest constrained negative obstacle detection and tracking with a stereo camera,” _IEEE Sensors Journal_ , vol. 22, no. 4, pp. 3616–3625, 2022.
  * [9] R. Fan, U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas, “Pothole detection based on disparity transformation and road surface modeling,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 897–908, 2020.
  * [10] Y. He and Z. Liu, “A feature fusion method to improve the driving obstacle detection under foggy weather,” _IEEE Transactions on Transportation Electrification_ , vol. 7, no. 4, pp. 2505–2515, 2021.
  * [11] J. Dib, K. Sirlantzis, and G. Howells, “A review on negative road anomaly detection methods,” _IEEE Access_ , vol. 8, pp. 57 298–57 316, 2020.
  * [12] P. Sun, X. Zhao, Z. Xu, R. Wang, and H. Min, “A 3d lidar data-based dedicated road boundary detection algorithm for autonomous vehicles,” _IEEE Access_ , vol. 7, pp. 29 623–29 638, 2019.
  * [13] G. Wang, J. Wu, R. He, and B. Tian, “Speed and accuracy tradeoff for lidar data based road boundary detection,” _IEEE/CAA Journal of Automatica Sinica_ , vol. 8, no. 6, pp. 1210–1220, 2021.
  * [14] G. Wang, J. Wu, R. He, and S. Yang, “A point cloud-based robust road curb detection and tracking method,” _IEEE Access_ , vol. 7, pp. 24 611–24 625, 2019.
  * [15] L. Chen, J. Yang, and H. Kong, “Lidar-histogram for fast road and obstacle detection,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 1343–1348.
  * [16] A. Dosovitskiy, G. Ros, F. Codevilla, A. M. López, and V. Koltun, “CARLA: An open urban driving simulator,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , 2017, pp. 1–16.
  * [17] M.-K. Choi, J.-H. Lee, H. Jung, I. R. Tayibnapis, and S. Kown, “Simulation framework for improved ui/ux of ar-hud display,” in _2018 IEEE International Conference on Consumer Electronics (ICCE)_ , 2018, pp. 1–4.
  * [18] N. Karatas, T. Tanaka, K. Fujikakc, Y. Yoshihara, H. Kanamori, Y. Fuwamoto, and M. Yoshida, “Evaluation of ar-hud interface during an automated intervention in manual driving,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_ , 2020, pp. 2158–2164.
  * [19] A. Dhiman and R. Klette, “Pothole detection using computer vision and learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 8, pp. 3536–3550, 2020.
  * [20] I. Schiopu, J. P. Saarinen, L. Kettunen, and I. Tabus, “Pothole detection and tracking in car video sequence,” in _2016 39th International Conference on Telecommunications and Signal Processing (TSP)_ , 2016, pp. 701–706.
  * [21] A. Akagic, E. Buza, and S. Omanovic, “Pothole detection: An efficient vision based method using rgb color space image segmentation,” in _2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_ , 2017, pp. 1104–1109.
  * [22] W. Sultani, S. Mokhtari, and H. Yun, “Automatic pavement object detection using superpixel segmentation combined with conditional random field,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 7, pp. 2076–2085, 2018.
  * [23] K. Azhar, F. Murtaza, M. H. Yousaf, and H. A. Habib, “Computer vision based detection and localization of potholes in asphalt pavement images,” in _2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE)_ , 2016, pp. 1–5.
  * [24] Y. Pan, X. Zhang, G. Cervone, and L. Yang, “Detection of asphalt pavement potholes and cracks based on the unmanned aerial vehicle multispectral imagery,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 11, no. 10, pp. 3701–3712, 2018\.
  * [25] H. Maeda, Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata, “Road damage detection and classification using deep neural networks with smartphone images,” _Computer-Aided Civil and Infrastructure Engineering_ , vol. 33, no. 12, pp. 1127–1141, 2018.
  * [26] Q. Mei, M. Gül, and M. R. Azim, “Densely connected deep neural network considering connectivity of pixels for automatic crack detection,” _Automation in Construction_ , vol. 110, p. 103018, 2020.
  * [27] F. Yang, L. Zhang, S. Yu, D. Prokhorov, X. Mei, and H. Ling, “Feature pyramid and hierarchical boosting network for pavement crack detection,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 4, pp. 1525–1535, 2019.
  * [28] Y. Shi, L. Cui, Z. Qi, F. Meng, and Z. Chen, “Automatic road crack detection using random structured forests,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 17, no. 12, pp. 3434–3445, 2016.
  * [29] S. Gu, Y. Zhang, J. Yang, and H. Kong, “Lidar-based urban road detection by histograms of normalized inverse depths and line scanning,” in _2017 European Conference on Mobile Robots (ECMR)_ , 2017, pp. 1–6.
  * [30] G. Bosurgi, M. Modica, O. Pellegrino, and G. Sollazzo, “An automatic pothole detection algorithm using pavement 3d data,” _International Journal of Pavement Engineering_ , pp. 1–15, 2022.
  * [31] F. M. Ortiz, M. Sammarco, L. H. M. Costa, and M. Detyniecki, “Vehicle telematics via exteroceptive sensors: A survey,” _arXiv preprint arXiv:2008.12632_ , 2020.
  * [32] G. Arvanitis, A. S. Lalos, and K. Moustakas, “Saliency mapping for processing 3d meshes in industrial modeling applications,” in _2019 IEEE 17th International Conference on Industrial Informatics (INDIN)_ , vol. 1, 2019, pp. 683–686.
  * [33] G. Arvanitis, A. S. Lalos, and K. Moustakas, “Robust and fast 3-d saliency mapping for industrial modeling applications,” _IEEE Transactions on Industrial Informatics_ , vol. 17, no. 2, pp. 1307–1317, 2021\.
  * [34] L. Ma and J. Li, “Sd-gcn: Saliency-based dilated graph convolution network for pavement crack extraction from 3d point clouds,” _International Journal of Applied Earth Observation and Geoinformation_ , vol. 111, p. 102836, 2022.
  * [35] H. Wang, H. Luo, C. Wen, J. Cheng, P. Li, Y. Chen, C. Wang, and J. Li, “Road boundaries detection based on local normal saliency from mobile laser scanning data,” _IEEE Geoscience and remote sensing letters_ , vol. 12, no. 10, pp. 2085–2089, 2015.
  * [36] T.-H. Wang, S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun, “V2vnet: Vehicle-to-vehicle communication for joint perception and prediction,” in _Computer Vision – ECCV 2020_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 605–621.
  * [37] E. Arnold, M. Dianati, R. de Temple, and S. Fallah, “Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 3, pp. 1852–1864, 2022.
  * [38] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, and J. Ma, “Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication,” 2021.
  * [39] Q. Chen, S. Tang, Q. Yang, and S. Fu, “Cooper: Cooperative perception for connected autonomous vehicles based on 3d point clouds,” in _2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)_. Los Alamitos, CA, USA: IEEE Computer Society, jul 2019, pp. 514–524. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICDCS.2019.00058
  * [40] Y.-C. Liu, J. Tian, N. Glaser, and Z. Kira, “When2com: Multi-agent perception via communication graph grouping,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 4105–4114.
  * [41] Q. Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu, “F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds,” in _Proceedings of the 4th ACM/IEEE Symposium on Edge Computing_ , ser. SEC ’19. New York, NY, USA: Association for Computing Machinery, 2019, p. 88–100. [Online]. Available: https://doi.org/10.1145/3318216.3363300
  * [42] J. Guo, D. Carrillo, S. Tang, Q. Chen, Q. Yang, S. Fu, X. Wang, N. Wang, and P. Palacharla, “Coff: Cooperative spatial feature fusion for 3-d object detection on autonomous vehicles,” _IEEE Internet of Things Journal_ , vol. 8, no. 14, pp. 11 078–11 087, 2021.
  * [43] Y. Yuan, H. Cheng, and M. Sester, “Keypoints-based deep feature fusion for cooperative vehicle detection of autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 3054–3061, 2022.
  * [44] S. Fang, H. Li, and M. Yang, “Lidar slam based multivehicle cooperative localization using iterated split cif,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–11, 2022.
  * [45] S.-W. Kim and W. Liu, “Cooperative autonomous driving: A mirror neuron inspired intention awareness and cooperative perception approach,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 8, no. 3, pp. 23–32, 2016.
  * [46] W. Song, Y. Yang, M. Fu, F. Qiu, and M. Wang, “Real-time obstacles detection and status classification for collision warning in a vehicle active safety system,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 19, no. 3, pp. 758–773, 2018.
  * [47] U. Ju, L. L. Chuang, and C. Wallraven, “Acoustic cues increase situational awareness in accident situations: A vr car-driving study,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 4, pp. 3281–3291, 2022.
  * [48] J. A. Abbasi, D. Mullins, N. Ringelstein, P. Reilhac, E. Jones, and M. Glavin, “An analysis of driver gaze behaviour at roundabouts,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–10, 2021.
  * [49] Q. Rao, C. Grünler, M. Hammori, and S. Chakraborty, “Design methods for augmented reality in-vehicle infotainment systems,” _Proceedings - Design Automation Conference_ , 06 2014.
  * [50] D. Strayer, J. M. Cooper, R. Goethe, M. M. McCarty, D. J. Getty, and F. N. Biondi, “Assessing the visual and cognitive demands of in-vehicle information systems,” _Cognitive Research: Principles and Implications_ , vol. 4, 2019.
  * [51] H. Kim and J. Gabbard, “Assessing distraction potential of augmented reality head-up displays for vehicle drivers,” _Human Factors The Journal of the Human Factors and Ergonomics Society_ , 05 2019.
  * [52] M. Dikmen and C. Burns, “Trust in autonomous vehicles: The case of tesla autopilot and summon,” in _2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)_ , 2017, pp. 1093–1098.
  * [53] D. Niu, J. Terken, and B. Eggen, “Anthropomorphizing information to enhance trust in autonomous vehicles,” _Human Factors and Ergonomics in Manufacturing & Service Industries_, vol. 28, 07 2017.
  * [54] L. Morra, F. Lamberti, F. G. Pratticò, S. Rosa, and P. Montuschi, “Building trust in autonomous vehicles: Role of virtual reality driving simulators in hmi design,” 07 2020.
  * [55] P. Lindemann, T. Lee, and G. Rigoll, “Supporting driver situation awareness for autonomous urban driving with an augmented-reality windshield display,” in _2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_ , 2018, pp. 358–363.
  * [56] A. O. Yontem, K. Li, D. Chu, V. Meijering, and L. Skrypchuk, “Prospective immersive human-machine interface for future vehicles: Multiple zones turn the full windscreen into a head-up display,” _IEEE Vehicular Technology Magazine_ , vol. 16, no. 1, pp. 83–92, 2021.
  * [57] H. Kim, J. L. Gabbard, A. M. Anon, and T. Misu, “Driver behavior and performance with augmented reality pedestrian collision warning: An outdoor user study,” _IEEE Transactions on Visualization and Computer Graphics_ , vol. 24, no. 4, pp. 1515–1524, 2018.
  * [58] Asam opendrive. [Online]. Available: https://www.asam.net/standards/detail/opendrive/
  * [59] P. Bazilinskyy, A. Eriksson, S. Petermeijer, and J. de Winter, “Usefulness and satisfaction of take-over requests for highly automated driving,” 10 2017.
  * [60] H. White, D. Large, D. Salanitri, G. Burnett, A. Lawson, and E. Box, “Rebuilding drivers’ situation awareness during take-over requests in level 3 automated cars,” 04 2019.
  * [61] K. Zeeb, A. Buchner, and M. Schrauf, “What determines the take-over time? an integrated model approach of driver take-over after automated driving,” _Accident; analysis and prevention_ , vol. 78, pp. 212–221, 03 2015.
  * [62] V. Melcher, S. Rauh, F. Diederichs, H. Widlroither, and W. Bauer, “Take-over requests for automated driving,” _Procedia Manufacturing_ , vol. 3, pp. 2867–2873, 2015, 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2351978915007891
  * [63] R. Fan, U. Ozgunalp, Y. Wang, M. Liu, and I. Pitas, “Rethinking road surface 3d reconstruction and pothole detection: From perspective transformation to disparity map segmentation,” _IEEE Transactions on Cybernetics_ , 2021.
  * [64] R. Fan, U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas, “Pothole detection based on disparity transformation and road surface modeling,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 897–908, 2019.
  * [65] R. Fan and M. Liu, “Road damage detection based on unsupervised disparity map segmentation,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 11, pp. 4906–4911, 2019.
  * [66] R. Fan, X. Ai, and N. Dahnoun, “Road surface 3d reconstruction based on dense subpixel disparity map estimation,” _IEEE Transactions on Image Processing_ , vol. 27, no. 6, pp. 3025–3035, 2018.
  * [67] Z. Zhang, X. Ai, C. K. Chan, and N. Dahnoun, “An efficient algorithm for pothole detection using stereo vision,” in _2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2014, pp. 564–568.
  * [68] U. Ozgunalp, X. Ai, and N. Dahnoun, _Signal, Image and Video Processing_ , vol. 10, no. 6, p. 1127–1134, Sep. 2016.
  * [69] P. Lindemann, T.-Y. Lee, and G. Rigoll, “Supporting driver situation awareness for autonomous urban driving with an augmented-reality windshield display,” in _2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_ , 2018, pp. 358–363.
  * [70] P. Lindemann, N. Müller, and G. Rigolll, “Exploring the use of augmented reality interfaces for driver assistance in short-notice takeovers,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_ , 2019, pp. 804–809.
  * [71] A. Riegler, A. Riener, and C. Holzmann, “Content presentation on 3d augmented reality windshield displays in the context of automated driving,” in _2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_ , 2022, pp. 543–552.
  * [72] E. J. Candès, X. Li, Y. Ma, and J. Wright, “Robust principal component analysis?” _Journal of the ACM (JACM)_ , vol. 58, no. 3, p. 11, 2011.
  * [73] Z. Lin, M. Chen, and Y. Ma, “The augmented lagrange multiplier method for exact recovery of corrupted low-rank matrices,” _CoRR_ , vol. abs/1009.5055, 2009.
  * [74] P. Rodríguez and B. Wohlberg, “Fast principal component pursuit via alternating minimization,” in _2013 IEEE International Conference on Image Processing_ , Sep. 2013, pp. 69–73.