the assessment module, the simplest measure is whether a real collision has
occurred or not, and the criticality definition also considered SOI.
Therefore, it was also implementation-specific and related to a collision. The
definitions in papers [74] and [109] do not consider a specific SoI. [74]
defined a scenario risk index and [109] used RNN for criticality definition
based on the collected data in simulation, and both are therefore non-
implementation-specific.

####  6.3.3 Mechanisms for Coverage

Here we summarize and present our observations on how these articles evaluate
the coverage results of the critical scenarios detected in their papers, all
of which are based, of course, on the fact that the articles contain
references to relevant content. Different methods also have different
mechanisms for coverage definitions and evaluations. For all articles that use
clustering methods, their coverage is based on the completeness of the feature
vectors (pre-defined numerical features that represent some scenario variables
or attributes), i.e., if these feature vectors are complete, then their
clustered typical scenarios can well reflect and cover the scenario space
represented by the defined scenario variables or attribute. The methods based
on existing data (textual analysis and reconstruction&simulation) do not
explicitly discuss the coverage results of the critical scenarios they found.

####  6.3.4 Required Information

All the required information in this section relates to sensor data,
unstructured accident records, structured accident records, sentence template
for NLP, simulator, FTA analysis, pre-crash scenario classes, traffic data and
real driving sensor data. The involved databases are Korea National Police
Agency Accident Database, NHTSA (National Highway Traffic Safety
Administration), IGLAD[155] (Initiative for the Global Harmonization of
Accident Data), Second Strategic Highway Research Program (SHRP2), China In-
Depth Accident Study (CIDAS) and German In-Depth Accident Study (GIDAS).

###  6.4 Deductive Reasoning

This section focuses on the methods that use a deductive reasoning approach
based on different sources of knowledge to find critical functional or logical
scenarios. Based on this scope, a total of 6 related articles were identified.

Four of these papers [61, 30, 62, 63] focus on finding pre-crash scenarios by
systematically considering all the possibilities under a set of explicitly
pre-defined assumptions. The identified pre-crash scenarios can be used as
safety-critical operational situations as introduced in Fig. 3. These four
papers adopt a similar approach. As the first step, they take some available
logical scenarios from function/system specification or safety analysis. Then,
they define a structured road and initial conditions based on the function
specification (e.g., 3 lane highway road) for the reasoning. Afterwards, the
scenario is elaborated using the assumptions, e.g., by adding more vehicles
and changing the vehicles’ behaviors. The assumptions include: potential
behaviors of vehicles on the road defined in the ODD, possible collision
types, traffic rules, and function (SoI) features. All the identified pre-
crash scenarios should be used for system verification and validation.

In addition to finding pre-crash scenarios as critical operational situations,
there is also an approach to find complete critical scenarios, which also
include a triggering condition. The authors in [64] proposed a method to
identify and to quantify the risk of critical scenarios for highly automated
driving vehicles, considering both functional insufficiencies and failures.
They proposed a new method to combine triggering conditions with the fault
tree to deduce critical scenario from a given safety goal. The approach can be
summarized into the following steps: The first step is to simulate the
automated driving functions as components and interfaces. The second step is
to identify potential hazards related to the AD system. Using a HAZOP-like
method, they first identify generic vehicle-level hazards that are independent
of the underlying implementation. Second, they use a HAZOP-like approach again
to identify Functional Insufficiencies with hazardous effects. Each AD
function is treated as a black box. Next, an “environmental fault tree” is
used to identify the causes of functional inadequacies and the corresponding
environmental conditions for triggering a hazardous scenario.

Ponn et al. [111] proposed a methodology for an intelligent selection of
relevant scenarios for the certification of automated vehicles. They proposed
a two-stage optimization framework to generate concrete scenarios. A detailed
optimization method was not proposed. In the first optimization stage, the
parameters of Layer one, two and five (refer to the 6-layered model discussed
in Sec. 2.3.1) are first optimized by sensor analysis and consideration of
driving behavior. In addition, the trajectory of the potential conflict
partner (Layer L4) is determined. In the second stage, further objects are
defined (to refine the logical scenario) by considering the complexity, and
their trajectories are optimized.

Criticality assessment: The criticality definitions of the articles in this
section are all shown in TABLE VI. All the articles in this section relate to
collision and are non-implementation specific. No implementation of a specific
system is needed for these approaches. Basically, a critical scenario should
contain at least one accident threat or collision (i.e., one accident type may
potentially happen).

####  6.4.1 Mechanisms for Coverage

The coverage in relation to the methods of this section depends on a set of
explicitly predefined assumptions in the scenario complexity definition (e.g.,
potential behaviors of vehicles on the road defined in ODD, possible collision
types, traffic rules, and function (SoI) features).

####  6.4.2 Required Information

The required information in this section also relates to the assumptions, such
as potential behaviors of vehicles on the highway, possible collision types,
traffic rules, set of base pre-crash scenarios, set of predefined basic
maneuvers, set of generic hazardous scenarios, function features, and system
specification.

###  6.5 Finding Critical Scenes for CV-based Functions

This section focuses on the cluster of methods to find critical scenes to
falsify a CV-based function. Based on the taxonomy in Fig. 10, methods in this
cluster are summarized in Fig. 24 and elaborated in the following subsections.

Figure 24: Methods to find critical scene for CV-based function

####  6.5.1 Scene Representation

In this cluster, scenes can be represented in the following three ways:

Scene as an image: To align with the terms defined in Fig. 4, an executable
scene is an image derived from a camera or stored in a database. Some methods
in this cluster directly generate critical scenes as images [113, 68, 121,
112]. Some methods take images as inputs to explore for critical ones [123,
122].

Description-based: In this representation, a scene is described by a set of
parameters [123, 122, 68, 124]. It can be used for both logical scenes and
concrete scenes. A logical scene contains the ranges or the probability
distributions of all the parameters. A concrete scene has a fixed value for
each parameter. A tool is needed to transform a concrete scene into an image,
based on, for instance, a video game (e.g., GTA V [68]) or a physics engine
(e.g., Unreal engine [112]).

Image plus transformation: This is used to represent a transformed image.
Examples of image transformation include shearing, blurring, adding
occlusions, and changing weather conditions [116]. With this representation, a
concrete scene is a transformed image, while a logical scenario is represented
as an original image together with a parameterized transformation (e.g.,
shearing with different angles) [116, 115].

Different scene representations are required by different methods to find or
to synthesize critical scenes. These methods are introduced in the next sub-
section.

####  6.5.2 Critical Scene Generation or Exploration

As shown in Fig. 24, we identified the following three major types of methods
to generate or explore a critical scene to falsify CV-based functions.

Image transformation: This type of methods assumes that a non-critical scene
can be made more critical by changing or adding content on the image, for
example, adding occlusions, intensifying the brightness, or changing weather
into extreme conditions. Our literature search found the following image
transformation approaches:

  * •

Direct image transformation: In this method, the image transformation is
applied directly to the image without the help of ML algorithms or
optimization algorithm. In general, direct image transformation (e.g.,
translation, scaling, shearing, rotation) [115]is a simple technique. However,
these transformations do not make the image more critical. They just generate
different images, which in most of the cases are not realistic. Image
transformation can also be performed by particular image processing tools. For
example, Yang et al. [112] used Adobe Photoshop to add blur or rain effect
through convolutional image transformation.

  * •

GAN method: A Generative Adversarial Network (GAN) can be trained to generate
new images that are similar but specifically different from the dataset [156].
In the context of CSI, GAN can help to automatically generate a large amount
of driving scenes with various weather conditions, which look realistic [121,
113].

  * •

Optimization-based method: Optimization algorithms can guide the exploration
towards critical scenes, and also increase the coverage. For example, Pei et
al. [115, 116] propose an optimization-based method to find a set of
combinations of direct image transformations to increase the neuron coverage
(i.e., the proportion of the neurons activated by a set of test inputs).
Increasing neuron coverage improves the quality of a test suite [157].

Finding critical scene from database: This type of methods traverses all the
images in a database to find the non-implementation-specific critical ones
[124, 122, 123]. Images in the databases are firstly manually encoded into
predefined parameterized scene descriptions. Thereafter, the core of this type
of methods is the approach to assess the criticality of each scene according
to its scene description. The criticality can be assessed based on the
complexity of the scene using a pre-trained machine learning (ML) model.
Discussion on criticality assessment methods in this cluster is given in
Section 6.5.3.

Generating critical scene according to scene descriptions: This type of
methods generates scenes according to a parameterized scene description with
predefined semantics and syntax. The scene description may include
environmental information such as weather conditions, light conditions and
time. It may also include the information of road users such as the relative
position, heading, dimension, car model, color, etc. According to these scene
descriptions, images are generated with the help of a third-party tool, such
as GTA V [68] and the Unreal Engine [112]. The generated scenes can support
data augmentation (see examples in [68]), the verification of CV-based
functions, and the identification of influential factors. Similar to the image
transformation method, this method can also minimize the effort and budget
compared to physically collecting these data. For instance, it can generate a
potentially critical scene by adding specified occlusions, different weather
conditions, or poor illumination.

####  6.5.3 Criticality Assessment

In this cluster, all the identified critical scenes are function-critical.
They may cause unintended results of CV-based functions. Most papers associate
critical scenes with challenging environment situations such as bad weather,
occlusions, and various light conditions. The criticality can be assessed
either directly on an image [68, 112, 113, 114, 124, 116, 115, 124], or on a
scene description [122, 123]. Surrogate measures to assess the criticality of
scenes are summarized in TABLE VI.

For the methods that directly assess criticality on images, most of them feed
the image under assessment to the SoI (i.e., the function under verification)
as input to assess its performance. Therefore, the assessed criticality is
implementation-specific. In TABLE VI, surrogate measures for these methods
include failures and differential behaviors, which are explained at the end of
this sub-section. In this literature review, only one paper [124] was found to
assess non-implementation-specific criticality on images. The corresponding
surrogate measure for this method [124] in TABLE VI is predictability, which
is also explained at the end of this sub-section. It is categorized as non-
implementation-specific since the criticality assessment process does not
involve the SoI.

As discussed previously, a scene description is a list of parameters, each of
which supports the description of one influential factor. Complexity in TABLE
VI is the surrogate measure to access the criticality of a scene based on all
the parameter values in its description. Complexity intends to reflect how
likely the performance of CV-based functions in general will be affected by
this scene. Therefore, complexity is categorized as non-implementation-
specific. Detailed explanation of complexity is elaborated at the end of this
subsection.

The detail explanation of each surrogate measure in TABLE VI for this cluster
is as follows:

  * •

Failures: It can help to find scenes that may lead the CV-based function to a
specific failure mode (e.g., mis-detection or mis-classification) [68, 112,
113, 114]. This measure is used to find consequence-aware implementation-
specific critical scenes.

  * •

Differential behaviors: This category is used to assess the Deep Learning (DL)
algorithms by measuring the number of differential behaviors (i.e., different
outputs from multiple similar DL systems with the same input)[116] [115]. In
safety-critical systems such as automated driving systems, differential
behavior can lead to disastrous consequences (e.g., collision, crash) [116].

  * •

Complexity: This category measures how challenging the scene is for the SoI.
In the primary studies, a complex scene can be specified according to project
experience [68, 121]. The complexity of a scene can also be evaluated by a
trained machine learning model. [122, 123]

  * •

Predictability: For one frame in a video stream, its criticality can be
assessed based on how difficult it can be predicted according to its previous
frames. Bolte et al. [124] utilized an image prediction algorithm to predict
each frame and compared the predicted frame with the real frame. The big
difference implies low predictability. In their method, images with relevant
objects (e.g., vehicles and pedestrians) and low predictability are considered
critical.

####  6.5.4 Mechanisms for Coverage

Coverage is rarely discussed in the primary studies in this cluster. The only
explicitly mentioned coverage is the neuron coverage (i.e., the proportion of
neurons activated by the whole test set) [116], which assesses the test
adequacy of neural networks. [157] This coverage is implementation-specific,
and can not reflect the completeness of the identified scenes. A coverage
defined based on the ODD can be a potential future research topic.

####  6.5.5 Required Information

We have identified general required information when generating or finding
critical scene for CV-based function as follows:

Scene generator: The scene generator is used for description-based or image
transformation-based method to generate a new scene. The scene generator can
be semantic-descriptor tools, game editor (Unreal) [112], photo editor (Adobe)
[116], etc.

Database: Most of the methods in this cluster required database. A critical
scene can be synthesized according to an existing scene in a database [121,
115, 116, 113]. A database can also be used to train the machine learning
model to access criticality of a new scene [122, 123].

Influential factors: The influential factors are required when specifying a
critical scene description [68, 112, 114].

##  7 The Evaluation Category

This section mainly focuses on the evaluation category defined in Section 4 to
answer the question ”How are the validity of the approach and the identified
critical scenarios assessed?” for each cluster defined in Section 6.

C1: Exploring logical scenarios without state trajectories
Almost all the primary studies in this cluster perform case studies as the
evaluation method to validate their proposed methods and the identified
critical scenarios. Most of these case studies are realized by simulation. The
validation in these papers shows that critical scenarios can be found using
the proposed methods based on the defined criticality metrics. However, in
these papers, there are no validations on the criticality of the scenarios
themselves. In other words, these articles did not tackle whether the
identified critical scenario in simulation is also critical in the real world,
or the gap between real world and simulation. Regarding coverage evaluation,
as shown in Fig. 20, different mechanisms like N-wise or sampling size were
proposed in this cluster, but there was no validation of the coverage results
in real-world interpretation.

C2: Exploring logical scenarios with state trajectories
The evaluation methods in this category are almost all case studies and very
similar to the ones of category C1.

C3: Inductive reasoning methods
Since the papers in this category used either existing databases or available
scenarios as data sources, the most used evaluation method is comparing the
identified scenarios with the scenarios in the regulations. In the papers that
generate functional or logical pre-crash scenarios by using clustering from
real data, they directly compare these functional scenarios with the test
scenario of the regulations or testing organizations, e.g., Euro NCAP ([110,
56, 54]). For other papers that generated concrete scenarios, the simulation
results, like simulated damage or KPIs using the proposed methods, were
compared directly with the recorded scenario or the test results. There was no
validation of the coverage results of the identified critical scenarios.

C4: Deductive reasoning methods
Not all the papers in this section performed a validation of the proposed
methods. One interesting validation method used in this cluster, however, is
presented in [61]. The authors used different knowledge, like possible
collision types and traffic rules, to generate the pre-crash functional
scenario and further evaluated the generated scenario catalog with respect to
the naturalistic driving database and test regulations (Euro Ncap, UNECE,
etc.). The coverage in this section was mostly based on the defined
assumptions of the scenario definition, and there was no validation of the
coverage.

C5: Finding critical scenes for CV-based functions
Almost all the articles in this category used a case study as the evaluation
method. Similar to the papers in C1 and C2, there are no validations made to
determine whether the identified critical scenario in simulation is also
critical in the real world for the computer vision system. For this cluster,
as mentioned in 6.5, the coverage depends mainly on the used dataset. No
validation of the coverage results has been found in this cluster.

##  8 Relation to Other Relevant Directions

In this section, we briefly present the other relevant directions discovered
during the literature review but are out of scope of the survey. These
relevant directions have important implications and relationships to the main
idea of the CSI method. In the following section, we first give a brief
introduction to these methods, then explain how they relate to our article,
and finally list the differences.

###  8.1 Online risk assessment

Online risk assessment refers to the methods which evaluate and predict in
real time if a situation could be dangerous and result in harm. As summarized
in [158], these methods analyze the predicted trajectories of other vehicles
and the potential colliding point. The methods presented in section 6.1.3 are
mainly offline methods, whereas the methods here are for real-time vehicle
application.

Relation: For both online and offline approaches, the identified influential
factors as well as the critical scenario based on unexpected behaviors or
events can be used. Second, the end-to-end scene risk prediction used in
online risk assessment can also be used offline to determine a scene’s
criticality. Furthermore, risk indicators such as Time-To-Collision and Time-
To-React can be used for both methods.

Differences: The online methods can only utilize the real street data whereas
the CSI methods can utilize ground-truth environmental information. Offline
methods, on the other hand, are not so easily transferred to online methods
because the offline approach typically requires exhaustive searches, and
computing all the potential trajectories of the vehicles is computationally
costly hens not real-time capable.

###  8.2 Scenario-based Function Assessment

Function assessment is one of the main reasons for using scenarios in the
development, verification and validation of ADSs. It is also one of the
primary reasons for finding critical scenarios as discussed throughout this
paper. Here we can consider both broad descriptions of scenarios, e.g., [36,
159], used for functional assessment, as well as different ways to achieve
coverage of the scenario space in general, e.g., through Monte-Carlo (MC)
simulations. Furthermore, [87] suggests a method using a heuristic search and
a fitness function to ensure that the scenarios specified are actually tested.
It is also possible to do functional assessment using MC simulations of agent
and simulation models [160]. In [160] these models are parameterized with
different data sources to reflect actual driving scenarios.

Relation: As mentioned, function assessment is a common vein across many of
the papers considered in this review, and one of the primary purposes for
using scenarios in general is to achieve some kind of assessment of the
function. This paper primarily focuses on the assessment of critical scenarios
whereas [36, 159, 87, 160] consider scenarios in a broader sense of AD
function assessment. Having a ”driver’s license test” for ADSs would perhaps
also be considered a function assessment. This aspect is however not
considered in this review nor further in this section.

Differences: The primary difference to the approaches of this section is the
lack of explicit discussion about how to find or identify critical scenarios.
In [36, 159] scenarios are used as important descriptors to achieve function
assessment, but there are no discussions about how to identify critical ones.
Ensuring that specified scenarios are indeed tested, as described in [87], is
surely relevant, but again outside of the scope of this review as [87]
disclose no details with respect to finding critical scenarios. One could
however note that the described methods could be used to ensure that critical
as well as non-critical scenarios that are specified are executed. Monte-Carlo
simulation, in general, is not covered in this review as MC only provides a
brute force methodology for covering scenario space. However, importance
sampling, as suggested in [161], is covered since it can be used to find the
scenarios that are more critical to the system. The authors of [161] have also
considered using subset simulation for accelerated testing of ADSs [162].
Similar to importance sampling, subset simulation guides the MC search for
relevant scenarios using a KPI. If this KPI is chosen to capture a safety-
critical aspect, this approach would also yield critical scenarios.

###  8.3 Scenario-based System Design

Scenarios have also been suggested for supporting system design. Authors in
[163] outline a method to identify critical spots of scenario space (white
spots) through simulation of an ADAS feature. These white spot areas are
subsequently used to design the ADAS system such that it fulfills the
requirements derived from the white spots identified in previous performance
assessment(s). The paper [164] in a related vein, suggests the inverse, namely
to define the ODD of the ADS using a world model of the intended use case for
the ADS, where this world model is proposed to be constructed based on
scenarios and populated using real traffic data. Paper [165] on the other hand
suggests the opposite again, that is, to use simulation assessment as a means
to determine the ODD of the ADS by removing road segments where the ADS
perform poorly (c.f. white spots of [163]).

Relation: In all three cases above, scenarios are used (implicitly) to support
the proposed process. Noteworthy is that both [165, 163] consider the parts of
scenario space (in their respective definitions) that result in critical
situations for the system being assessed.

Differences: Even though scenarios are considered, neither of the approaches
discuss how to explicitly find the critical scenarios (white spots of [163]),
which is the primary focus of this review. Thus, deploying the methods
described in this review would increase the efficiency of [165, 163] in terms
of needed resources for simulations.

###  8.4 Fault Injection

Fault Injection (FI) is a method that accelerates the occurrence of faults to
test and evaluate faults in systems. It is an important part of the test
process in the automotive standard ISO 26262 [16] and is distributed over the
development stages. The author in [21] applied the fault injection method to
find and mine situations and faults that can be dangerous to the vehicle’s
safety.

Relation: The target of article [21] is to find safety-critical situation and
faults by using causal and counter-factual reasoning about the behavior of the
ADS under a fault for safety analysis. FI method can therefore also be used to
find critical scenario and situation.

Differences: The primary goal of FI is to evaluate the faults in system and
analyze the system behavior based on the injected faults. The basic purpose of
FI differs from that of the CSI method, but it is an interesting and well-
established method for testing the system in autonomous driving applications.

###  8.5 Ontology Design and Influential Factor Analysis

The articles and methods here focus on finding and analyzing the influencing
factor, and utilizing a domain ontology to capture and represent the
environment of the system under test. As presented in Fig 3, a scenario
condition can be modeled as a combination of several scenario factors. The
main steps of the ontology design method in the section are similar to the
method in the reasoning section regarding both deductive and inductive
reasoning. They took input or knowledge from different sources, such as system
specification, sensor error type, system knowledge or analysis like FTA, or
FMEA analysis [166].

Relation: The articles in this section are not included in the main content.
The reason is that they focus mainly on the use of ontology design to find
triggering conditions or influencing factors, but these identified influencing
factors can be further used in CSI method to find the critical scenarios.

Differences: The main difference between this section’s approaches is that no
further case study or application were performed to identify critical
scenarios.

###  8.6 Formal Methods

Formal methods are used to define and verify unambiguous specifications of
computer systems. These methods rely on a sufficiently complete abstract model
of the real world and formal specifications that should resemble the informal
requirements that should be imposed on the system. [167]

Relation: This literature review covers the papers which use formal methods to
find counter examples as critical scenarios.

Differences: It is necessary to represent the relations between the behaviors
of the agents and environmental conditions with hybrid automata, which need to
be sophisticated enough to reflect reality and simple enough to guarantee
computational feasibility.

###  8.7 Unknown Unknowns Detection in Computer Vision

Unknown unknowns in computer vision (CV) functions (e.g., object
classification) refer to the case where the employed predictive model (e.g.,
an deep neural network) assign incorrect labels to instances with high
confidence [168]. Explainable AI [169] approaches can be adopted to detect and
avoid unknown unknowns.

Relation: These unknown unknowns are typically caused by the mismatch between
training data and the cases encountered at test time. Since the CV functions
are almost blind to such errors, they can be considered as implementation-
specific critical scenes.

Differences: According to the authors’ understanding, these approaches should
be considered in Section 6.5. However, no automated driving applications of
these approaches were found during our literature search.

###  8.8 Data Augmentation

Data augmentation is used for machine learning based functions to expand the
size of the training set by generating new training data.

Relation: Section 6.5 covers methods to generate new scenes according to a
predefined scene description. These generated scenes can also be used for data
augmentation.

Differences: Traditional data augmentation methods focus on how to generate a
big amount of different scenes as training data. Methods introduced in Section
6.5 focus on generating new and also potentially critical scenes.

##  9 Discussion

This section discusses how the results of the literature review (as presented
in Sections 4, 5, 6 and 7) answer the three research questions defined in
Section 3.1, repeated below.

  1. RQ1:

What would be a taxonomy that allows to systematically categorize and compare
state-of-the art CSI methods for ADS and ADAS?

  2. RQ2:

What is the current status of CSI methods research with respect to this
taxonomy?

  3. RQ3:

What are the remaining problems and challenges for further investigation?

We have addressed RQ1 by analyzing relevant industrial standards, as discussed
in Section 2, and the primary studies collected with the methodology
introduced in Section 3. The taxonomy was derived through iterations and
validated by applying it to the collected studies. The proposed taxonomy, as
presented in Section 4, can sufficiently categorize all the primary studies.
To support further research and engineering in areas related to critical
scenarios and their identification methods, this taxonomy can help researchers
to provide a contextual understanding - the big picture of such methods-, and
provide guidance regarding CSI method design and adoption.

RQ2 is answered in Sections 5, 6 and 7 by analyzing all the primary studies
according to the taxonomy. The answer to this question is also used as the
basis to answer RQ3.

RQ3 is answered by internal discussions among all the authors; ”remaining
problems and challenges” are presented in Sections 5, 6 and 7. Some important
points are summarized and discussed in the rest of this section, including
suggestions for future work.

###  9.1 Coverage

Within the context of this paper, coverage can be defined in 3 ways (referred
to as coverage types in the following): 1) the coverage of the exploration
with respect to the given scenario space; 2) the coverage of all the critical
scenarios in the given scenario space (i.e., the proportion of the identified
critical scenarios among all the critical scenarios within the given scenario
space); and 3) the coverage of all the critical functional insufficiencies and
their triggering conditions under a given functional scenario or an ODD. The
type 1 coverage evaluates the exploration of the scenario space. The type 2
coverage evaluates the effectiveness of a logical $\rightarrow$ concrete CSI
method. As shown in Fig. 6, identified critical scenarios are used to support
the identification of critical functional insufficiencies so as to improve the
safety of the intended functionalities. Therefore, the type 3 coverage is
essential for safety analysis. These three types of coverage are discussed in
the rest of this section.

The type 1 coverage is valid for the CSI methods finding critical concrete
scenarios from a given logical scenario. Most of these methods are covered in
clusters C1, C2 and C5. Due to the huge size of the scenario space, reaching
full coverage is practically impossible. To quantify the level of coverage,
coverage metrics can be defined. Adopted coverage metrics include sampling
size (for sample-based exploration methods) and combinatorial coverage (for
combinatorial testing methods). It is difficult to directly measure the type 2
and type 3 coverage. The type 2 coverage can be indirectly reflected by the
coverage metrics defined for the type 1 coverage. As discussed before, the
type 3 coverage is of the most importance. However, the relationships between
these three types of coverage have not been sufficiently discussed in the
primary studies. Understanding these relationships can help to determine a
sufficient coverage level (in terms of a particular coverage metric) on the
scenario space. It can also help to analyze the completeness of the safety
analysis. These relationships can be discussed from the perspective of how
critical scenarios may be missed.

Relation between type 1 and type 2 coverage: With potential limitations of an
adopted exploration method, it may not be possible to identify all the
critical concrete scenarios in a given scenario space (i.e., the given logical
scenario). According to Fig. 16, these limitations can stem from the coverage
level employed by the instantiation process or the criticality assessment
method. Since full coverage is not reached, some critical scenarios may not be
covered by the specified coverage level. Even though a critical scenario is
reached during the exploration, it may still be assessed as noncritical due to
the limitation of the criticality assessment method. If a simulator is used
for criticality assessment, its fidelity (i.e. the influential factors in the
simulator can represent those factors in real world) will affect the accuracy
of the criticality assessment. Increasing the fidelity of simulators is one
way to solve this problem. However, a high-fidelity simulator normally entails
high computational resources, and thereby increases power and time
consumption. Another way to solve this problem is to design surrogate
criticality measures that can tolerate some simulation error (i.e. the
differences between simulation and real results). For example, instead of
collision, Time To Collision (TTC) can be used to find more potentially
critical scenarios. However, as analyzed in Section 6.5.3 and shown in Table
VI, no such surrogate measure was found in the primary studies to find
critical scenes for the CV methods. On the other hand, some critical scenarios
may also be filtered out by the limitation of the adopted surrogate
criticality measure. For example, if longitudinal TTC is used as the surrogate
measure, critical scenarios caused by lateral collision will be missed.
Therefore an appropriate approach to determine the criticality measure can be
another future research direction. In addition, another way to increase the
fidelity of the simulator is to bring the vehicle into the loop of the
simulation, such as the methods introduced in [94] and [82].

Relation between type 2 and type 3 coverage: Even if all the critical concrete
scenarios within the given logical scenario are identified, one can still not
claim a full coverage of the type 3 coverage under the corresponding
functional scenario. Due to the potential misalignment between the logical
scenario and the functional scenario, there might be critical scenarios that
are not covered by the logical scenario. The reason for the misalignment is
the assumptions made when formalizing the functional scenario (referring to
Fig. 4). For example, some influential factors might be missing, or the models
of some influential factors might be too simplified (e.g., using a constant
speed model to represent the behavior of another vehicle). Section 8.5 briefly
discusses how to find and formulate influential factors. Even though it is not
the focus of this survey paper, it is an important topic for the safety
analysis in terms of SOTIF. A literature review on influential factor
identification could be a valuable future work.

Instead of being identified from critical scenarios, functional
insufficiencies can also be derived by analyzing the AD functions with safety
analysis methods, such as the ones introduced in [170] and [64]. As inspired
by traditional safety engineering [171], the coverage of the derived
functional insufficiencies can be increased by conducting both top-down (e.g.,
fault tree analysis) and bottom-up (e.g., Failure Modes and Effects Analysis
(FMEA)) safety analysis methods, where the top refers to the vehicle level
safety goals, and the bottom refers to the malfunctioning behaviors of the
components together with the corresponding triggering conditions. To this end,
a potential valuable future research direction could be to propose a
methodology that combines these safety analysis approaches and the CSI
approaces introduced in this paper.

In addition, as discussed in Section 6.1.4, some primary studies assume that
close (in terms of the distance on the scenario space) critical scenarios are
likely to reflect the same functional insufficiency with the same triggering
condition. To this end, increasing the diversity of the identified critical
scenarios can increase the coverage of the identified functional
insufficiencies. However, explicit coverage metrics for the identified
functional insufficiencies and triggering conditions have not been found in
the primary studies.

When systematically deducing safety-critical operational situations (methods
in cluster C4), the coverage is defined by the assumptions made about the
possible behaviors of the involved traffic participants (including the ego
vehicle) and the possible road topology. In the primary studies, these
assumptions are made on layers L1 and L4 of the 6-layer scenario description
model in Table II, since they all consider highway scenarios. For urban
scenarios, the layer L2 in the 6-layer model (e.g., traffic lights) should
also be considered. In addition, as discussed in Section 7, coverage of the
deduced safety-critical operational situations can be evaluated against an
accident database.

Corresponding to Fig 3, to achieve a complete safety analysis, coverage needs
to be explicitly defined for hazards, malfunctioning behaviors, triggering
conditions and safety-critical operational situations. As a precondition, this
entails an explicit coverage definition for all the influential factors. These
factors are classified into layers as explained in Table II. A more detailed
classification of these factors in each layer is necessary to facilitate the
systematic analysis of both safety-critical operational situations and
triggering conditions. Therefore, this would represent another relevant future
research topic.

###  9.2 The ALARP Principle

Identifying all the functional insufficiencies with all the triggering
conditions is practically impossible. The iteration in Fig. 6 should be
governed by the ALARP principle (i.e. to guarantee that the risk of harm is
”As Low As Reasonably Practicable”), see e.g., [171]. The core concept behind
”reasonably practicable” is to control the risk of harm to an expected level.
Therefore it is necessary to know how to control risk; and how to determine
the expected level.

ISO 26262 [16] answers these two questions for the risk caused by systematic
faults and random hardware faults. In ISO 26262, the risk of harm is
determined by the severity and the probability of the harm. The ALARP
principle in ISO 26262 is to control the residual risk (the risk remaining
after the deployment of all the safety measures) to a reasonable level.

For the harm caused by functional insufficiency, risk can be defined based on
the severity of the harm and the probability of exposure of the safety-
critical scenario. In some of the primary studies, this probability of
exposure is estimated according to the historical traffic data of human-driven
vehicles. However, automated driving vehicles may exhibit different driving
behaviors. Therefore, human-driving data may not precisely reflect the
situations with automated driving vehicles. To this end, an effective approach
to estimate the risk of harm related to functional insufficiencies represents
a future research direction. In particular, given the novelty and unknowns of
automated driving, this emphasizes the important role of continuous data
gathering to support methodology improvements. An ability to better estimate
the risk of harm would can support the determination of the stop condition of
the iterative process illustrated in Fig. 6.

###  9.3 Scenario Space Explosion

Due to the openness of the driving environment, a vast amount of influential
factors compose an ODD to explore. The exploration effort can be reduced by
”divide and conquer” approaches. To our understanding, the scenario space can
be partitioned from two perspectives.

As suggested in [164], the first perspective is to divide an ODD into
applicable use cases. Each use case is a sub-space of the ODD represented as
one functional scenario. In this way, critical scenarios are identified for
each use case, instead of the whole ODD. The challenge of this method is to
guarantee that the union set of all the use cases is equivalent to the whole
ODD.

Different SoIs provide a second perspective for partitioning a scenario space
(i.e. a logical ODD). As discussed in [15], triggering conditions for
different automated driving functions (i.e. SoIs) contain different
influential factors. For example, the angle of the sun may affect some
perception functions, but it will not affect the vehicle dynamic control
functions. Therefore, the number of influential factors for a particular AD
function is much lower than that for the whole ADS. To this end, the
exploration effort of a CSI method can be reduced by finding triggering
conditions for each individual AD function. The identified triggering
conditions for all the AD functions can be systematically combined to support
the final safety analysis on the vehicle level. However, the propagation of a
functional insufficiency is not deterministic due to the resilience of the
downstream AD functions [21]. For example, if a vehicle is not detected in
some individual frames by the computer vision function, this can be resolved
by the following object tracing and sensor fusion algorithms for most cases.
Therefore, to have a complete safety analysis, it is necessary to explicitly
analyze the resilience of the system (i.e. to identify the scenarios where
unintended behaviors from upstream functions cannot be resolved), especially
to analyze the resilience of the object tracking functions and the sensor
fusion functions. According to this literature review, only one paper [57] was
found to indirectly touch this topic. Investigating resilience as just
described, would thus provide a useful research direction. Moreover, a
systemic view would in addition be required to ensure that the partitioning
into the SoIs - and the ensuing composition of ”evidence” is complete, for
example with respect to common cause failures. As discussed in this section
this requires new methods that combine different approaches to critical
scenarios identification, coverage and validation.

###  9.4 Other Sources of Harm

Fig. 3 shows the sources of harm that are considered in this paper. Besides
these, there are also other aspects that can threaten the safety of an ADS,
such as the ones listed below. For a complete safety analysis, all the sources
of harm should be considered together, since a combination of multiple sources
of harm may lead to a new hazardous event.

Potential lack of Communication: As shown in [172], some functional
insufficiencies cannot be resolved without V2V (Vehicle to Vehicle) and V2I
(Vehicle to Infrastructure) communication. Examples of such functional
insufficiencies include occlusions, traffic violation by other road users, and
the uncertainty of behavior prediction.

Cyberattack: An ADS can be hacked, especially when V2X (vehicle to everything)
communication is adopted. Standard ISO/SAE 21434 [14] discusses how to
mitigate the risk regarding cybersecurity.

Misuse: Misuse is considered in ISO/PAS 21448 [10] but not emphsized in this
paper. It can be considered as part of a scenario on layer L0 in Table II.

Problems outside the embedded system: ISO 26262 [16] and ISO/PAS 21448 [10]
only focus the the safety issues caused by the embedded system. Other vehicle
failures, such as a flat tire, a broken suspension or a battery fire, can also
lead to significant harms.

##  10 Conclusion

For assuring safety of autonomous and automated driving, it is essential to be
able to efficiently and effectively derive critical scenarios, i.e. situations
that cause potential risks of harm (safety risks). Such critical scenarios
need explicit consideration in ADS design and V&V efforts. Moreover, the use
of state of the art techniques for CSI promises to reduce the V&V space by
focusing on critical scenarios.

In this paper, we contribute to the challenge of safety and quality assurance
of ADS and ADAS. In particular, we presented the results obtained from a
systematic literature review in the area of CSI methods. Moreover, we
introduced a taxonomy focusing on practitioners for supporting them in
selecting the right CSI method for identifying critical scenarios according to
their project-specific needs. The introduced survey covers assumptions, levels
of abstraction, metrics stating criticality, methods for the generation of
critical scenarios and coverage measures.

In addition, we discussed research gaps obtained after analyzing the primary
studies, for example the coverage metrics, the criticality measures, the
identification and classification of influential factors, and the lack of a
methodological framework to combine different CSI methods and also to connect
the CSI methods with other safety analysis processes. Hence this survey may
also be considered helpful in providing guidance regarding future research
directions.

## Acknowledgments

The authors of this paper have received support from a number of initiatives
including: TECoSA - the Vinnova Competence Center for Trustworthy Edge
Computing Systems and Applications at KTH Royal Institute of Technology and
the ECSEL projects InSecTT (grant agreement No 876038 - www.insectt.eu) and
ArchitectECA2030 (grant agreement No 877539 - https://autoc3rt.automotive.oth-
aw.de/). The ECSEL projects have received funding from the ECSEL Joint
Undertaking which receives support from the European Union’s Horizon 2020
research and innovation programme and Austria, Sweden, Spain, Italy, France,
Portugal, Ireland, Finland, Slovenia, Poland, Netherlands, Turkey. Finally,
support is also acknowledged from the CD-Laboratory for Quality Assurance
Methodologies for Autonomous Cyber-Physical Systems (QAMCAS) (
https://qamcas.ist.tugraz.at/) with financial support by the Austrian Federal
Ministry for Digital and Economic Affairs, the National Foundation for
Research, Technology and Development and the Christian Doppler Research
Association.

## Disclaimer

The document reflects only the view of the authors. The European Commission
and the companies (i.e. Sigma Technology Consulting, Scania, AVL List GmbH, TU
Graz and Zenseact) are not responsible for any use that may be made of the
information it contains.

## References

  * [1] M. Törngren, “Cyber-physical systems have far-reaching implications,” _HiPEAC Vision 2021_ , Jan. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4710500
  * [2] E. Frazzoli, “Can we put a price on autonomous driving?” _MIT Technology Review_ , March 2014.
  * [3] C. F. Kerry and J. Karsten, “Gauging investments in self-driving cars,” 2017.
  * [4] A. Shetty, M. Yu, A. Kurzhanskiy, O. Grembek, H. Tavafoghi, and P. Varaiya, “Safety challenges for autonomous vehicles in the absence of connectivity,” _Transportation Research Part C: Emerging Technologies_ , vol. 128, p. 103133, 2021.
  * [5] N. Kalra and S. M. Paddock, “Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?” _Transportation Research Part A: Policy and Practice_ , vol. 94, pp. 182–193, 2016.
  * [6] On-Road Automated Driving (ORAD) committee, “J3016 Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles,” SURFACE VEHICLE RECOMMENDED PRACTICE, Tech. Rep., 2021.
  * [7] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a Formal Model of Safe and Scalable Self-driving Cars,” _arXiv:1708.06374 [cs, stat]_ , 2017, arXiv: 1708.06374.
  * [8] S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer, “Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving,” in _2015 IEEE 18th International Conference on Intelligent Transportation Systems_ , vol. 2015-Octob. IEEE, sep 2015, pp. 982–988.
  * [9] ASAM, “ASAM: STANDARDIZATION FOR HIGHLY AUTOMATED DRIVING SIM:Guide,” Association for Standardization of Automation and Measuring Systems, Tech. Rep., 2021. [Online]. Available: https://www.asam.net/asam-guide-simulation/
  * [10] International Organization for Standardization, “Road vehicles - Safety of the intended functionality,” 2019.
  * [11] Underwriters Laboratories, Northbrook, USA, “UL 4600:2020 – Standard for Evaluation of Autonomous Products,” 2020.
  * [12] IEEE Committee VT/ITS - Intelligent Transportation Systems, “P2846 - Assumptions for Models in Safety-Related Automated Vehicle Behavior - Under development,” 2021.
  * [13] “Iso/awi ts 5083: Road vehicles — safety for automated driving systems — design, verification and validation - under development,” 2021.
  * [14] “Iso/sae 21434 - road vehicles — cybersecurity engineering (under development),” 2021.
  * [15] C. Amersbach and H. Winner, “Functional decomposition—A contribution to overcome the parameter space explosion during validation of highly automated driving,” _Traffic Injury Prevention_ , vol. 20, no. sup1, pp. S52–S57, jun 2019.
  * [16] International Organization for Standardization, “ISO 26262: Road vehicles–Functional safety,” 2018.
  * [17] The British Standards Institution, “Operational Design Domain (ODD) taxonomy for an automated driving system (ADS) – Specification,” 2020.
  * [18] International Organization for Standardization, “Road vehicles — Terms and definitions of test scenarios for automated driving systems,” 2021.
  * [19] ——, “Road vehicles — Engineering framework and process of scenario-based safety evaluation,” 2021.
  * [20] ——, “Road vehicles — Taxonomy for operational design domain for automated driving systems,” 2021.
  * [21] S. Jha, S. Banerjee, T. Tsai, S. K. Hari, M. B. Sullivan, Z. T. Kalbarczyk, S. W. Keckler, and R. K. Iyer, “Ml-based fault injection for autonomous vehicles: A case for bayesian fault injection,” in _2019 49th annual IEEE/IFIP international conference on dependable systems and networks (DSN)_. IEEE, 2019, pp. 112–124.
  * [22] T. Menzel, G. Bagschik, and M. Maurer, “Scenarios for development, test and validation of automated vehicles,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 1821–1827.
  * [23] S. Kato, S. Tokunaga, Y. Maruyama, S. Maeda, M. Hirabayashi, Y. Kitsukawa, A. Monrroy, T. Ando, Y. Fujii, and T. Azumi, “Autoware on board: Enabling autonomous vehicles with embedded systems,” in _2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS)_. IEEE, 2018, pp. 287–296.
  * [24] S. Kato, E. Takeuchi, Y. Ishiguro, Y. Ninomiya, K. Takeda, and T. Hamada, “An Open Approach to Autonomous Vehicles,” _IEEE Micro_ , vol. 35, no. 6, pp. 60–68, nov 2015. [Online]. Available: http://ieeexplore.ieee.org/document/7368032/
  * [25] J. Xu, Q. Luo, K. Xu, X. Xiao, S. Yu, J. Hu, J. Miao, and J. Wang, “An automated learning-based procedure for large-scale vehicle dynamics modeling on baidu apollo platform,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 5049–5056.
  * [26] B. Giesler, “An Appeal Practical Standards in Autonomous Driving,” _ATZelektronik worldwide_ , vol. 12, no. 2, pp. 16–21, apr 2017.
  * [27] M. Törngren, X. Zhang, N. Mohan, M. Becker, L. Svensson, X. Tao, D.-J. Chen, and J. Westman, “Architecting safety supervisors for high levels of automated driving,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 1721–1728.
  * [28] J. M. Carroll, _Scenario-based design: envisioning work and technology in system development_. John Wiley & Sons, Inc., 1995.
  * [29] H. Winner, K. Lemmer, T. Form, and J. Mazzega, “Pegasus—first steps for the safe introduction of automated driving,” in _Road Vehicle Automation 5_. Springer, 2019, pp. 185–195.
  * [30] H. Weber, J. Bock, J. Klimke, C. Roesener, J. Hiller, R. Krajewski, A. Zlocki, and L. Eckstein, “A framework for definition of logical scenarios for safety assurance of automated driving,” _Traffic Injury Prevention_ , vol. 20, no. sup1, pp. S65–S70, jun 2019.
  * [31] P. Koopman, A. Kane, and J. Black, “Credible autonomy safety argumentation,” in _27th Safety-Critical Sys. Symp. Safety-Critical Systems Club, Bristol, UK_ , 2019.
  * [32] C. Neurohr, L. Westhofen, T. Henning, T. de Graaff, E. Möhlmann, and E. Böde, “Fundamental considerations around scenario-based testing for automated driving,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 121–127.
  * [33] S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” _IEEE access_ , vol. 8, pp. 87 456–87 477, 2020.
  * [34] S. Keele _et al._ , “Guidelines for performing systematic literature reviews in software engineering,” Citeseer, Tech. Rep., 2007.
  * [35] P. Brereton, B. A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” _Journal of systems and software_ , vol. 80, no. 4, pp. 571–583, 2007.
  * [36] J. Bach, S. Otten, and E. Sax, “Model based scenario specification for development and test of automated driving functions,” _IEEE Intelligent Vehicles Symposium, Proceedings_ , vol. 2016-Augus, no. Iv, pp. 1149–1155, 2016.
  * [37] H. A. Alhaija, S. K. Mustikovela, L. Mescheder, A. Geiger, and C. Rother, “Augmented reality meets computer vision: Efficient data generation for urban driving scenes,” _International Journal of Computer Vision_ , vol. 126, no. 9, pp. 961–972, 2018.
  * [38] M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen, and R. Vasudevan, “Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 746–753, 2017.
  * [39] A. Aleti, B. Buhnova, L. Grunske, A. Koziolek, and I. Meedeniya, “Software architecture optimization methods: A systematic literature review,” _IEEE Transactions on Software Engineering_ , vol. 39, no. 5, pp. 658–683, may 2013.
  * [40] X. Zhang, J. Tao, K. Tan, M. Törngren, J. M. Gaspar Sánchez, M. R. Ramli, X. Tao, M. Gyllenhammar, F. Wotawa, N. Mohan, M. Nica, and H. Felbinger, “Finding critical scenarios for automated driving systems: The data extraction form,” KTH, Tech. Rep., 2021. [Online]. Available: http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-302116
  * [41] Y. Li, J. Tao, and F. Wotawa, “Ontology-based test generation for automated and autonomous driving functions,” _Information and software technology_ , vol. 117, p. 106200, 2020.
  * [42] E. De Gelder and J. P. Paardekooper, “Assessment of Automated Driving Systems using real-life scenarios,” _IEEE Intelligent Vehicles Symposium, Proceedings_ , no. Iv, pp. 589–594, 2017.
  * [43] F. Reiterer, J. Zhou, J. Kovanda, V. Rulc, V. Kemka, and L. del Re, “Beyond-Design-Basis Evaluation of Advanced Driver Assistance Systems,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_ , jun 2019, pp. 2119–2124.
  * [44] Z. Huang, H. Lam, and D. Zhao, “Sequential experimentation to efficiently test automated vehicles,” in _2017 Winter Simulation Conference (WSC)_. IEEE, dec 2017, pp. 3078–3089.
  * [45] D. Zhao, H. Lam, H. Peng, S. Bao, D. J. LeBlanc, K. Nobukawa, and C. S. Pan, “Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques,” _IEEE transactions on intelligent transportation systems_ , vol. 18, no. 3, pp. 595–607, 2016.
  * [46] M. Koschi, C. Pek, S. Maierhofer, and M. Althoff, “Computationally Efficient Safety Falsification of Adaptive Cruise Control Systems,” _2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019_ , pp. 2879–2886, 2019\.
  * [47] N. Li, I. Kolmanovsky, and A. Girard, “Model-free optimal control based automotive control system falsification,” in _2017 American Control Conference (ACC)_ , may 2017, pp. 636–641.
  * [48] S. Khastgir, G. Dhadyalla, S. Birrell, S. Redmond, R. Addinall, and P. Jennings, “Test Scenario Generation for Driving Simulators Using Constrained Randomization Technique,” in _SAE Technical Paper_. sae.org, mar 2017.
  * [49] R. B. Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, “Testing vision-based control systems using learnable evolutionary algorithms,” _Proceedings of the 40th International Conference on Software Engineering - ICSE ’18_ , 2018.
  * [50] F. Kluck, M. Zimmermann, F. Wotawa, and M. Nica, “Genetic Algorithm-Based Test Parameter Optimization for ADAS System Testing,” in _2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, jul 2019, Conference Paper, pp. 418–425.
  * [51] F. Klück, M. Zimmermann, F. Wotawa, and M. Nica, “Performance comparison of two search-based testing strategies for adas system validation,” in _IFIP International Conference on Testing Software and Systems_. Springer, 2019, pp. 140–156.
  * [52] H. Beglerovic, M. Stolz, and M. Horn, “Testing of autonomous vehicles using surrogate models and stochastic optimization,” _IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC_ , vol. 2018-March, pp. 1–6, 2018.
  * [53] X. Wang, H. Peng, and D. Zhao, “Combining reachability analysis and importance sampling for accelerated evaluation of highway automated vehicles at pedestrian crossing,” _ASME Letters in Dynamic Systems and Control_ , vol. 1, no. 1, 2021.
  * [54] W. Hu, X. Xu, Z. Zhou, Y. Liu, Y. Wang, L. Xiao, and X. Qian, “Mining and comparative analysis of typical pre-crash scenarios from iglad,” _Accident Analysis & Prevention_, vol. 145, p. 105699, 2020.
  * [55] Y. Kim, S. Tak, J. Kim, and H. Yeo, “Identifying major accident scenarios in intersection and evaluation of collision warning system,” in _2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2017, pp. 1–6.
  * [56] B. Sui, N. Lubbe, and J. Bärgman, “A clustering approach to developing car-to-two-wheeler test scenarios for the assessment of automated emergency braking in china using in-depth chinese crash data,” _Accident Analysis & Prevention_, vol. 132, p. 105242, 2019.
  * [57] C. E. Tuncali, G. Fainekos, D. Prokhorov, H. Ito, and J. Kapinski, “Requirements-driven test generation for autonomous vehicles with machine learning components,” _IEEE Transactions on Intelligent Vehicles_ , vol. 5, no. 2, pp. 265–280, 2019.
  * [58] M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, “Adaptive Stress Testing for Autonomous Vehicles,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , vol. 2018-June. IEEE, jun 2018, pp. 1–7.
  * [59] A. Corso, P. Du, K. Driggs-Campbell, and M. J. Kochenderfer, “Adaptive stress testing with reward augmentation for autonomous vehicle validatio,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, 2019, pp. 163–168.
  * [60] M. Koren and M. J. Kochenderfer, “Efficient Autonomy Validation in Simulation with Adaptive Stress Testing,” in _2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019_. IEEE, oct 2019, pp. 4178–4183.
  * [61] J. Zhou and L. del Re, “Reduced complexity safety testing for adas & adf,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 5985–5990, 2017.
  * [62] L. Huang, Q. Xia, F. Xie, H.-L. Xiu, and H. Shu, “Study on the Test Scenarios of Level 2 Automated Vehicles,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, jun 2018, pp. 49–54.
  * [63] F. Xie, T. Chen, Q. Xia, L. Huang, and H. Shu, “Study on the Controlled Field Test Scenarios of Automated Vehicles,” in _SAE Technical Papers_ , vol. 2018-Augus, no. August, aug 2018.
  * [64] B. Kramer, C. Neurohr, M. Büker, E. Böde, M. Fränzle, and W. Damm, “Identification and quantification of hazardous scenarios for automated driving,” in _International Symposium on Model-Based Safety and Assessment_. Springer, 2020, pp. 163–178.
  * [65] C. E. Tuncali, G. Fainekos, H. Ito, and J. Kapinski, “Simulation-based adversarial test generation for autonomous vehicles with machine learning components,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2018, pp. 1555–1562.
  * [66] C. E. Tuncali and G. Fainekos, “Rapidly-exploring Random Trees for Testing Automated Vehicles,” _2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019_ , pp. 661–666, 2019.
  * [67] F. Batsch, A. Daneshkhah, M. Cheah, S. Kanarachos, and A. Baxendale, “Performance Boundary Identification for the Evaluation of Automated Vehicles using Gaussian Process Classification,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, oct 2019, pp. 419–424.
  * [68] D. Fremont, X. Yue, T. Dreossi, S. Ghosh, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, “Scenic: Language-based scene generation,” _arXiv preprint arXiv:1809.09310_ , 2018.
  * [69] M. Klischat and M. Althoff, “Generating critical test scenarios for automated vehicles with evolutionary algorithms,” _IEEE Intelligent Vehicles Symposium, Proceedings_ , vol. 2019-June, no. Iv, pp. 2352–2358, 2019.
  * [70] M. Althoff and S. Lutz, “Automatic Generation of Safety-Critical Test Scenarios for Collision Avoidance of Road Vehicles,” _IEEE Intelligent Vehicles Symposium, Proceedings_ , vol. 2018-June, no. Iv, pp. 1326–1333, 2018\.
  * [71] C. E. Tuncali, S. Yaghoubi, T. P. Pavlic, and G. Fainekos, “Functional gradient descent optimization for automatic test case generation for vehicle controllers,” in _2017 13th IEEE Conference on Automation Science and Engineering (CASE)_ , vol. 2017-Augus. IEEE, aug 2017, pp. 1059–1064.
  * [72] P. Koopman and F. Fratrik, “How Many Operational Design Domains, Objects, and Events?” in _Proceedings of AAAI Workshop on Artificial Intelligence Safety_ , Honolulu, USA, jan 2019.
  * [73] L. Stark, M. Düring, S. Schoenawa, J. E. Maschke, and C. M. Do, “Quantifying Vision Zero: Crash avoidance in rural and motorway accident scenarios by combination of ACC, AEB, and LKS projected to German accident occurrence,” _Traffic Injury Prevention_ , vol. 20, no. sup1, pp. S126–S132, 2019.
  * [74] B. Yue, S. Shi, S. Wang, and N. Lin, “Low-Cost Urban Test Scenario Generation Using Microscopic Traffic Simulation,” _IEEE Access_ , vol. 8, pp. 123 398–123 407, 2020.
  * [75] J. Tao, Y. Li, F. Wotawa, H. Felbinger, and M. Nica, “On the industrial application of combinatorial testing for autonomous driving functions,” in _2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)_. IEEE, 2019, pp. 234–240.
  * [76] F. Klück, Y. Li, M. Nica, J. Tao, and F. Wotawa, “Using ontologies for test suites generation for automated and autonomous driving functions,” in _2018 IEEE International symposium on software reliability engineering workshops (ISSREW)_. IEEE, 2018, pp. 118–123.
  * [77] G. E. Mullins, P. G. Stankiewicz, R. C. Hawthorne, and S. K. Gupta, “Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles,” _Journal of Systems and Software_ , vol. 137, pp. 197–215, mar 2018.
  * [78] M. Nabhan, M. Schoenauer, Y. Tourbier, and H. Hage, “Optimizing coverage of simulated driving scenarios for the autonomous vehicle,” in _2019 IEEE International Conference on Connected Vehicles and Expo (ICCVE)_. IEEE, nov 2019, pp. 1–5.
  * [79] Y. Akagi, R. Kato, S. Kitajima, J. Antona-Makoshi, and N. Uchida, “A Risk-index based Sampling Method to Generate Scenarios for the Evaluation of Automated Driving Vehicle Safety *,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, oct 2019, pp. 667–672.
  * [80] S. Feng, Y. Feng, C. Yu, Y. Zhang, and H. X. Liu, “Testing scenario library generation for connected and automated vehicles, part i: Methodology,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 3, pp. 1573–1582, 2020.
  * [81] S. Feng, Y. Feng, H. Sun, S. Bao, Y. Zhang, and H. X. Liu, “Testing scenario library generation for connected and automated vehicles, part ii: Case studies,” _IEEE Transactions on Intelligent Transportation Systems_ , 2020\.
  * [82] S. Feng, Y. Feng, X. Yan, S. Shen, S. Xu, and H. X. Liu, “Safety assessment of highly automated driving systems in test tracks: a new framework,” _Accident Analysis & Prevention_, vol. 144, p. 105664, 2020.
  * [83] S. Feng, Y. Feng, H. Sun, Y. Zhang, and H. X. Liu, “Testing scenario library generation for connected and automated vehicles: An adaptive framework,” _IEEE Transactions on Intelligent Transportation Systems_ , pp. 1–10, 2020\.
  * [84] A. Gambi, M. Mueller, and G. Fraser, “Automatically testing self-driving cars with search-based procedural content generation,” in _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_ , 2019, pp. 318–328.
  * [85] W. Huang, Y. Lv, L. Chen, and F. Zhu, “Accelerate the autonomous vehicles reliability testing in parallel paradigm,” in _2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2017, pp. 922–927.
  * [86] D. Stumper and K. Dietmayer, “Towards Criticality Characterization of Situational Space,” in _IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC_ , vol. 2018-Novem, 2018, pp. 3378–3382.
  * [87] F. Hauer, A. Pretschner, and B. Holzmüller, “Fitness functions for testing automated and autonomous driving systems,” in _International Conference on Computer Safety, Reliability, and Security_. Springer, 2019, pp. 69–84.
  * [88] S. Wagner, A. Knoll, K. Groh, T. Kühbeck, D. Watzenig, and L. Eckstein, “Virtual assessment of automated driving: Methodology, challenges, and lessons learned,” _SAE International Journal of Connected and Automated Vehicles_ , vol. 2, no. 12-02-04-0020, pp. 263–277, 2019.
  * [89] C. Gladisch, T. Heinz, C. Heinzemann, J. Oehlerking, A. von Vietinghoff, and T. Pfitzer, “Experience Paper: Search-Based Testing in Automated Driving Control Applications,” in _2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, nov 2019, pp. 26–37.
  * [90] P. Junietz, F. Bonakdar, B. Klamann, and H. Winner, “Criticality Metric for the Safety Validation of Automated Driving using Model Predictive Trajectory Optimization,” in _IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC_ , vol. 2018-Novem, 2018.
  * [91] G. Chance, A. Ghobrial, S. Lemaignan, T. Pipe, and K. Eder, “An Agency-Directed Approach to Test Generation for Simulation-based Autonomous Vehicle Verification,” in _2020 IEEE International Conference On Artificial Intelligence Testing (AITest)_. IEEE, aug 2020, pp. 31–38.
  * [92] S. Wagner, K. Groh, T. Kuhbeck, M. Dorfel, and A. Knoll, “Using Time-to-React based on Naturalistic Traffic Object Behavior for Scenario-Based Risk Assessment of Automated Driving,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , vol. 2018-June, no. Iv. IEEE, jun 2018, pp. 1521–1528.
  * [93] P. Du and K. Driggs-Campbell, “Finding Diverse Failure Scenarios in Autonomous Systems Using Adaptive Stress Testing,” _SAE International Journal of Connected and Automated Vehicles_ , vol. 2, no. 4, pp. 12–02–04–0018, dec 2019.
  * [94] P. Junietz, W. Wachenfeld, V. Schönemann, K. Domhardt, W. Tribelhorn, and H. Winner, “Gaining Knowledge on Automated Driving’s Safety—The Risk-Free VAAFO Tool,” in _Lecture Notes in Control and Information Sciences_. Springer International Publishing, 2019, vol. 476, pp. 47–65.
  * [95] B. Gangopadhyay, S. Khastgir, S. Dey, P. Dasgupta, G. Montana, and P. Jennings, “Identification of Test Cases for Automated Driving Systems Using Bayesian Optimization,” _2019 IEEE Intelligent Transportation Systems Conference, ITSC 2019_ , pp. 1961–1967, 2019.
  * [96] S. Kuutti, S. Fallah, and R. Bowden, “Training adversarial agents to exploit weaknesses in deep control policies,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 108–114.
  * [97] Y. Abeysirigoonawardena, F. Shkurti, and G. Dudek, “Generating adversarial driving scenarios in high-fidelity simulators,” _Proceedings - IEEE International Conference on Robotics and Automation_ , vol. 2019-May, pp. 8271–8277, 2019.
  * [98] M. O’Kelly, A. Sinha, H. Namkoong, J. Duchi, and R. Tedrake, “Scalable end-to-end autonomous vehicle testing via rare-event simulation,” in _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , ser. NIPS’18. Red Hook, NY, USA: Curran Associates Inc., 2018, p. 9849–9860.
  * [99] A. Gambi, T. Huynh, and G. Fraser, “Generating effective test cases for self-driving cars from police reports,” in _Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE 2019_. ACM Press, 2019, pp. 257–267.
  * [100] ——, “Automatically reconstructing car crashes from police reports for testing self-driving cars,” in _2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)_. IEEE, 2019, pp. 290–291.
  * [101] Y. Qi, K. Li, W. Kong, Y. Wang, and Y. Luo, “A trajectory-based method for scenario analysis and test effort reduction for highly automated vehicle,” _SAE Technical Papers_ , vol. 2019-April, no. April, pp. 1–8, 2019.
  * [102] X. Qin, N. Aréchiga, A. Best, and J. Deshmukh, “Automatic testing and falsification with dynamically constrained reinforcement learning,” _arXiv preprint arXiv:1910.13645_ , 2019.
  * [103] G. Chou, Y. E. Sahin, L. Yang, K. J. Rutledge, P. Nilsson, and N. Ozay, “Using control synthesis to generate corner cases: A case study on autonomous driving,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 37, no. 11, pp. 2906–2917, 2018.
  * [104] J. Zhou and L. del Re, “Safety Verification Of ADAS By Collision-free Boundary Searching Of A Parameterized Catalog,” in _2018 Annual American Control Conference (ACC)_ , jun 2018, pp. 4790–4795.
  * [105] S. Cutrone, C. W. Liew, B. Utter, and A. Brown, “A Framework for Identifying and Simulating Worst-Case Animal-Vehicle Interactions,” in _Proceedings - 2018 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2018_ , 2019, pp. 1995–2000.
  * [106] X. Ma, Z. Ma, X. Zhu, J. Cao, and F. Yu, “Driver Behavior Classification under Cut-In Scenarios Using Support Vector Machine Based on Naturalistic Driving Data,” in _WCX SAE World Congress Experience_. SAE International, apr 2019.
  * [107] V. Bithar and A. Karumanchi, “Application of collision probability estimation to calibration of advanced driver assistance systems,” _SAE Technical Papers_ , vol. 2019-April, no. April, 2019.
  * [108] S. Masuda, H. Nakamura, and K. Kajitani, “Rule-based searching for collision test cases of autonomous vehicles simulation,” _IET Intelligent Transport Systems_ , vol. 12, no. 9, pp. 1088–1095, 2018.
  * [109] I. R. Jenkins, L. O. Gee, A. Knauss, H. Yin, and J. Schroeder, “Accident Scenario Generation with Recurrent Neural Networks,” in _IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC_ , vol. 2018-Novem, 2018, pp. 3340–3345.
  * [110] J. J. So, I. Park, J. Wee, S. Park, and I. Yun, “Generating Traffic Safety Test Scenarios for Automated Vehicles using a Big Data Technique,” _KSCE Journal of Civil Engineering_ , vol. 23, no. 6, pp. 2702–2712, jun 2019\.
  * [111] T. Ponn, C. Gnandt, and F. Diermeyer, “An optimization-based method to identify relevant scenarios for type approval of automated vehicles,” in _Proceedings of the ESV—International Technical Conference on the Enhanced Safety of Vehicles, Eindhoven, The Netherlands_ , 2019, pp. 10–13.
  * [112] S. Yang, W. Deng, Z. Liu, and Y. Wang, “Analysis of illumination condition effect on vehicle detection in photo-realistic virtual world,” SAE Technical Paper, Tech. Rep., 2017.
  * [113] H. Yu and Xin Li, “Intelligent corner synthesis via cycle-consistent generative adversarial networks for efficient validation of autonomous driving systems,” in _2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC)_ , 2018, pp. 9–15.
  * [114] T. Dreossi, A. Donzé, and S. A. Seshia, “Compositional falsification of cyber-physical systems wit machine learning components,” _J. Autom. Reason._ , vol. 63, no. 4, pp. 1031–1053, 2019.
  * [115] Y. Tian, K. Pei, S. Jana, and B. Ray, “DeepTest: automated testing of deep-neural-network-driven autonomous cars,” in _Proceedings of the 40th International Conference on Software Engineering_. ACM, may 2018, pp. 303–314.
  * [116] K. Pei, Y. Cao, J. Yang, and S. Jana, “Deepxplore: Automated whitebox testing of deep learning systems,” in _proceedings of the 26th Symposium on Operating Systems Principles_ , 2017, pp. 1–18.
  * [117] F. Gao, J. Duan, Y. He, and Z. Wang, “A Test Scenario Automatic Generation Strategy for Intelligent Driving Systems,” _Mathematical Problems in Engineering_ , vol. 2019, pp. 1–10, jan 2019.
  * [118] Q. Xia, J. Duan, F. Gao, T. Chen, and C. Yang, “Automatic Generation Method of Test Scenario for ADAS Based on Complexity,” in _SAE Technical Papers_ , vol. Part F1298, no. September, sep 2017.
  * [119] Q. Xia, J. Duan, F. Gao, Q. Hu, and Y. He, “Test Scenario Design for Intelligent Driving System Ensuring Coverage and Effectiveness,” _International Journal of Automotive Technology_ , vol. 19, no. 4, pp. 751–758, aug 2018.
  * [120] J. Duan, F. Gao, and Y. He, “Test Scenario Generation and Optimization Technology for Intelligent Driving Systems,” _IEEE Intelligent Transportation Systems Magazine_ , p. 1, 2020.
  * [121] M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid, “Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,” in _2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 2018, pp. 132–142.
  * [122] J. Wang, C. Zhang, Y. Liu, and Q. Zhang, “Traffic sensory data classification by quantifying scenario complexity,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 1543–1548.
  * [123] C. Zhang, Y. Liu, Q. Zhang, and L. Wang, “A graded offline evaluation framework for intelligent vehicle’s cognitive ability,” in _2018 IEEE Intelligent Vehicles Symposium (IV)_ , 2018, pp. 320–325.
  * [124] J. Bolte, A. Bar, D. Lipinski, and T. Fingscheidt, “Towards corner case detection for autonomous driving,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_ , 2019, pp. 438–445.
  * [125] S. Hallerbach, Y. Xia, U. Eberle, and F. Koester, “Simulation-Based Identification of Critical Scenarios for Cooperative and Automated Vehicles,” _SAE International Journal of Connected and Automated Vehicles_ , vol. 1, no. 2, pp. 2018–01–1066, apr 2018.
  * [126] R.-D. Reiss, _Approximate distributions of order statistics: with applications to nonparametric statistics_. Springer science & business media, 2012.
  * [127] D. R. Kuhn, R. Bryce, F. Duan, L. S. Ghandehari, Y. Lei, and R. N. Kacker, “Combinatorial testing: Theory and practice,” _Advances in Computers_ , vol. 99, pp. 1–66, 2015.
  * [128] E.-H. Choi, T. Kitamura, C. Artho, and Y. Oiwa, “Design of prioritized n-wise testing,” in _IFIP International Conference on Testing Software and Systems_. Springer, 2014, pp. 186–191.
  * [129] C. Nie and H. Leung, “A survey of combinatorial testing,” _ACM Computing Surveys_ , vol. 43, no. 2, pp. 1–29, jan 2011.
  * [130] H. Liu, F. Hussain, C. L. Tan, and M. Dash, “Discretization: An enabling technique,” _Data mining and knowledge discovery_ , vol. 6, no. 4, pp. 393–423, 2002.
  * [131] M. J. Kochenderfer and T. A. Wheeler, _Algorithms for optimization_. Mit Press, 2019.
  * [132] J.-k. Xiao, W.-m. Li, W. Li, and X.-r. Xiao, “Optimization on black box function optimization problem,” _Mathematical Problems in Engineering_ , vol. 2015, 2015.
  * [133] K. Meinke, F. Niu, and M. Sindhu, “Learning-based software testing: a tutorial,” in _International Symposium On Leveraging Applications of Formal Methods, Verification and Validation_. Springer, 2011, pp. 200–219.
  * [134] P. McMinn, “Search-based software testing: Past, present and future,” in _2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops_. IEEE, 2011, pp. 153–163.
  * [135] C. Cotta, M. Sevaux, and K. Sörensen, _Adaptive and multilevel metaheuristics_. Springer, 2008, vol. 136\.
  * [136] P. M. Junietz, “Microscopic and macroscopic risk metrics for the safety validation of automated driving. Ph.D. thesis,” _TU Darmstadt, Darmstadt_ , 2019.
  * [137] S. M. Mahmud, L. Ferreira, M. S. Hoque, and A. Tavassoli, “Application of proximal surrogate indicators for safety evaluation: A review of recent developments and research needs,” _IATSS Research_ , vol. 41, no. 4, pp. 153–163, 2017.
  * [138] T. Ponn, D. Fratzke, C. Gnandt, and M. Lienkamp, “Towards Certification of Autonomous Driving: Systematic Test Case Generation for a Comprehensive but Economically-Feasible Assessment of Lane Keeping Assist Algorithms,” in _Proceedings of the 5th International Conference on Vehicle Technology and Intelligent Transport Systems_ , no. Vehits. SCITEPRESS - Science and Technology Publications, 2019, pp. 333–342.
  * [139] A. Corso, R. J. Moss, M. Koren, R. Lee, and M. J. Kochenderfer, “A Survey of Algorithms for Black-Box Safety Validation,” _arXiv_ , may 2020.
  * [140] P. I. Frazier, “A Tutorial on Bayesian Optimization,” _arXiv:1807.02811 [cs, math, stat]_ , Jul. 2018, arXiv: 1807.02811.
  * [141] S. M. LaValle _et al._ , “Rapidly-exploring random trees: A new tool for path planning,” 1998.
  * [142] R. Sargent, “Optimal control,” _Journal of Computational and Applied Mathematics_ , vol. 124, no. 1-2, pp. 361–371, 2000.
  * [143] C. Fan, U. Mathur, S. Mitra, and M. Viswanathan, “Controller synthesis made real: Reach-avoid specifications and linear dynamics,” in _Computer Aided Verification_ , H. Chockler and G. Weissenbacher, Eds. Cham: Springer International Publishing, 2018, pp. 347–366.
  * [144] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: a survey,” _Journal of Artificial Intelligence Research_ , vol. 4, no. 1, pp. 237–285, May 1996.
  * [145] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [146] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A Survey of Monte Carlo Tree Search Methods,” _IEEE Transactions on Computational Intelligence and AI in Games_ , vol. 4, no. 1, pp. 1–43, Mar. 2012, conference Name: IEEE Transactions on Computational Intelligence and AI in Games.
  * [147] S. S. Mousavi, M. Schukat, and E. Howley, “Deep reinforcement learning: An overview,” in _Proceedings of SAI Intelligent Systems Conference (IntelliSys) 2016_ , Y. Bi, S. Kapoor, and R. Bhatia, Eds. Cham: Springer International Publishing, 2018, pp. 426–440.
  * [148] R. Lee, M. J. Kochenderfer, O. J. Mengshoel, G. P. Brat, and M. P. Owen, “Adaptive stress testing of airborne collision avoidance systems,” in _2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC)_ , 2015, pp. 6C2–1–6C2–13.
  * [149] A. Censi, K. Slutsky, T. Wongpiromsarn, D. Yershov, S. Pendleton, J. Fu, and E. Frazzoli, “Liability, ethics, and culture-aware behavior specification using rulebooks,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 8536–8542.
  * [150] NHTSA, “2016 fatal motor vehicle crashes: Overview,” 2017.
  * [151] V. an der TU Dresden GmbH, “Codebook GIDAS2017,” 2018.
  * [152] J. Bakker, H. Jeppsson, L. Hannawald, F. Spitzhüttl, A. Longton, and E. Tomasch, “Iglad-international harmonized in-depth accident data,” in _Proceedings of the 25th International Conference on Enhanced Safety of Vehicles (ESV). Detroit, MI_ , 2017, pp. 1–12.
  * [153] P. Nitsche, P. Thomas, R. Stuetz, and R. Welsh, “Pre-crash scenarios at road junctions: A clustering method for car crash data,” _Accident Analysis & Prevention_, vol. 107, pp. 137–151, 2017.
  * [154] P. M. Bhagat, P. S. Halgaonkar, and V. M. Wadhai, “Review of clustering algorithm for categorical data,” _International Journal of Engineering and Advanced Technology_ , vol. 3, no. 2, 2013.
  * [155] Q. Chen, Y. Chen, O. Bostrom, Y. Ma, and E. Liu, “A comparison study of car-to-pedestrian and car-to-e-bike accidents: data source: the china in-depth accident study (cidas),” SAE Technical Paper, Tech. Rep., 2014.
  * [156] Z. Pan, W. Yu, X. Yi, A. Khan, F. Yuan, and Y. Zheng, “Recent progress on generative adversarial networks (gans): A survey,” _IEEE Access_ , vol. 7, pp. 36 322–36 333, 2019.
  * [157] F. Harel-Canada, L. Wang, M. A. Gulzar, Q. Gu, and M. Kim, “Is neuron coverage a meaningful measure for testing deep neural networks?” in _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , ser. ESEC/FSE 2020\. New York, NY, USA: Association for Computing Machinery, 2020, p. 851–862.
  * [158] S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion prediction and risk assessment for intelligent vehicles,” _ROBOMECH Journal_ , vol. 1, no. 1, p. 1, dec 2014.
  * [159] C. Roesener, F. Fahrenkrog, A. Uhlig, and L. Eckstein, “A scenario-based assessment approach for automated driving by using time series classification of human-driving behaviour,” in _2016 IEEE 19th international conference on intelligent transportation systems (ITSC)_. IEEE, 2016, pp. 1360–1365.
  * [160] L. Wang, F. Fahrenkrog, T. Vogt, O. Jung, and R. Kates, “Prospective safety assessment of highly automated driving functions using stochastic traffic simulation,” in _25th International Technical Conference on the Enhanced Safety of Vehicles (ESV) National Highway Traffic Safety Administration, Detroit_ , 2017.
  * [161] D. Zhao, H. Peng, H. Lam, S. Bao, K. Nobukawa, D. J. LeBlanc, and C. S. Pan, “Accelerated evaluation of automated vehicles in lane change scenarios,” in _Dynamic Systems and Control Conference_ , vol. 57243. American Society of Mechanical Engineers, 2015, p. V001T17A002.
  * [162] S. Zhang, H. Peng, D. Zhao, and H. E. Tseng, “Accelerated evaluation of autonomous vehicles in the lane change scenario based on subset simulation technique,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 3935–3940.
  * [163] L. Stark, S. Obst, S. Schoenawa, and M. Düring, “Towards vision zero: Addressing white spots by accident data based adas design and evaluation,” in _2019 IEEE International Conference on Vehicular Electronics and Safety (ICVES)_. IEEE, 2019, pp. 1–6.
  * [164] M. Gyllenhammar, R. Johansson, F. Warg, D. Chen, H.-M. Heyn, M. Sanfridson, J. Söderberg, A. Thorsén, and S. Ursing, “Towards an operational design domain that supports the safety argumentation of an automated driving system,” in _10th European Congress on Embedded Real Time Systems (ERTS 2020)_ , 2020.
  * [165] C. W. Lee, N. Nayeer, D. E. Garcia, A. Agrawal, and B. Liu, “Identifying the operational design domain for an automated driving system through assessed risk,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 1317–1322.
  * [166] P. Cao and L. Huang, “Application oriented testcase generation for validation of environment perception sensor in automated driving systems,” SAE Technical Paper, Tech. Rep., 2018.
  * [167] J. Wing, “A specifier’s introduction to formal methods,” _Computer_ , vol. 23, no. 9, pp. 8–22, 1990.
  * [168] H. Lakkaraju, E. Kamar, R. Caruana, and E. Horvitz, “Identifying unknown unknowns in the open world: Representations and policies for guided exploration,” in _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , ser. AAAI’17. AAAI Press, 2017, p. 2124–2132.
  * [169] F. K. Došilović, M. Brčić, and N. Hlupić, “Explainable artificial intelligence: A survey,” in _2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_ , 2018, pp. 0210–0215.
  * [170] O. Zendel, M. Murschitz, M. Humenberger, and W. Herzner, “Cv-hazop: Introducing test data validation for computer vision,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 2066–2074.
  * [171] N. G. Leveson, _Engineering a safer world: Systems thinking applied to safety_. The MIT Press, 2016.
  * [172] A. Shetty, M. Yu, A. Kurzhanskiy, O. Grembek, H. Tavafoghi, and P. Varaiya, “Safety challenges for autonomous vehicles in the absence of connectivity,” _Transportation Research Part C: Emerging Technologies_ , vol. 128, no. June 2017, p. 103133, jul 2021.