11institutetext: FZI Research Center for Information Technology, Germany
11email<EMAIL_ADDRESS>
22institutetext: Karlsruhe Institute of Technology, Germany

# One Ontology to Rule Them All:
Corner Case Scenarios for Autonomous Driving

Daniel Bogdoll* 1122 Stefani Guneshka* 22 J. Marius Zöllner 1122

###### Abstract

The core obstacle towards a large-scale deployment of autonomous vehicles
currently lies in the long tail of rare events. These are extremely
challenging since they do not occur often in the utilized training data for
deep neural networks. To tackle this problem, we propose the generation of
additional synthetic training data, covering a wide variety of corner case
scenarios. As ontologies can represent human expert knowledge while enabling
computational processing, we use them to describe scenarios. Our proposed
master ontology is capable to model scenarios from all common corner case
categories found in the literature. From this one master ontology, arbitrary
scenario-describing ontologies can be derived. In an automated fashion, these
can be converted into the OpenSCENARIO format and subsequently executed in
simulation. This way, also challenging test and evaluation scenarios can be
generated.

###### Keywords:

corner cases, ontology, scenarios, synthetic data, simulation, autonomous
driving

††* These authors contributed equally

##  1 Introduction

For selected Operational Design Domains (ODD), autonomous vehicles of the SAE
level 4 [33] can already be seen on the roads [44]. However, it remains highly
debated, how the safety of these vehicles can be shown and steadily improved
in a structured way. In Germany, the first country with a federal law for
level 4 autonomous driving, the safety of such vehicles needs to be
demonstrated based on a catalog of test scenarios [19, 18]. However, the
coverage of rare, but highly relevant corner cases [27] in scenario-based
descriptions poses a significant challenge [35]. Data-driven, learned scenario
generation approaches currently tend to focus on adversarial scenarios with a
high risk of collision [14, 43, 21, 37], neglecting other forms of corner
cases. While there exist comprehensive taxonomies on the types and categories
of corner cases [9, 22], there exist no generation method tailored to these
most important long-tail scenes and scenarios. Based on this, the verification
and validation during testing and ultimately the scalability of autonomous
driving systems to larger ODDs in real world deployments remain enormous
challenges. To tackle these challenges, it is necessary to generate a large
variety of rare corner case scenarios for the purposes of training, testing,
and evaluation of autonomous vehicle systems. As shown by Tuncali et al. [41],
model-, data-, and scenario-based methods can be used for this purpose. An
extensive overview on these can be found in [6]. However, the authors find
that, ”while there are knowledge-based descriptions and taxonomies for corner
cases, there is little research on machine-interpretable descriptions” [6].

To fill this gap between knowledge- and data-driven approaches for the
description and generation of corner case scenarios111We follow the
definitions of scene and scenario by Ulbrich et al. [42], where a scene is a
snapshot, and a scenario consists of successive scenes, we propose the first
scenario generation method which is capable of generating all corner case
categories, also called levels, described by Breitenstein et al. [9] in a
scalable fashion, where all types of scenarios can be derived from a single
master ontology. Based on the resulting scenario-describing ontologies,
synthetic data of rare corner case scenarios can be generated automatically.
This newly created training data hopefully contributes to an increased
robustness of deep neural networks to anomalies, helping make deep neural
networks safer. For a general introduction to ontologies in the domain of
information sciences, we refer to [32]. More details can be found in [20]. All
code and data is available on GitHub.

The remainder of this work is structured as follows: In Section 2, we provide
an overview of related ontology-based scene and scenario description methods
and outline the identified research gap. In Section 3, we describe how our
proposed master ontology is designed and the automated process to design and
generate scenario ontologies from it. In Section 4, we demonstrate how
different, concrete scenarios can be derived from the master ontology and how
the resulting ontologies can be used to execute these in simulation. Finally,
in Section 5, we provide a brief summary and outline next steps and future
directions.

##  2 Related Work

While there exist many ways of describing scenarios [6], ontologies are the
most powerful way of doing so, as these are not only human- and machine-
readable, but also extremely scalable for the generation of scenarios, when
used in the right fashion [23]. Ontologies are being widely used for the
description of scenarios. In the work of Bagschik et al. [5], an ontology is
presented which describes simple highway scenarios based on a set of pre-
defined keywords. In a later work, Menzel et al. [30] extend the concept to
generate OpenSCENARIO and OpenDRIVE scenarios, while many of the relevant
details were not modelled in the ontology itself, but in post-processing
steps. For the description of the surrounding environment of a vehicle, Fuchs
et al. [16] especially focus on lanes and occupying traffic participants,
while neglecting their actions. Li et al. [29] also create scenarios which are
executed in a simulation environment, covering primarily situations, where
sudden braking maneuvers are necessary. Thus, their ontology is very domain-
specific. They build upon their previous works [40, 45, 28]. Tahir and
Alexander [39] propose an ontology that focuses on intersections due to their
high collision rates. They show that their scenarios can be executed in
simulation, while focusing on changing weather conditions. While they claim to
have developed an ontology, the released code [46] only contains scripted
scenarios, which might be derived from an ontology structurally. Hermann et
al. [23] propose an ontology for dataset creation, with a demonstrated focus
on pedestrian detection, including pedestrian occlusions. Their ontology is
structurally inspired by the Pegasus model [38] and consists of 22 sub-
ontologies. It is capable of describing a wide variety of scenarios and
translate them into simulation. However, since the ontology itself is neither
described in detail nor publicly available, it does not become clear whether
each frame requires a separate ontology or whether the ontology itself is able
to describe temporal scenarios. In the OpenXOntology project by ASAM [3], an
ontology is being developed with the purpose to unify their different
products, such as OpenSCENARIO or OpenDRIVE. Based on the large body of
previous work in the field of scenario descriptions, this ontology is
promising for further developments. However, at the moment, it serves the
purpose of a taxonomy. Finally, Gelder et al. [17] propose an extensive
framework for the development of a ”full ontology of scenarios”. At the
moment, they have not developed the ontology itself yet, which is why their
work cannot be compared to existing ontologies.

Authors | Year |  | Temporal Scenario
---
Description
| Arbitrary
---
Environments
| Arbitrary
---
Objects
| Scenario
---
Simulation
| Corner Case
---
Categorization
| Ontology
---
available
Fuchs et al. [16] | 2008 | - | - | ✓ | - | - | -
Hummel [25] | 2010 | - | ✓ | - | - | - | -
Hülsen et al. [26] | 2011 | ✓ | ✓ | - | - | - | -
Armand et al. [1] | 2014 | - | - | - | - | - | -
Zhao et al. [47] | 2017 | - | ✓ | - | - | - | ✓
Bagschik et al. [5] | 2018 | ✓ | - | - | - | - | -
Chen and Kloul [13] | 2018 | ✓ | - | - | - | - | -
Huang et al. [24] | 2019 | ✓ | - | - | - | - | -
Menzel et al. [30] | 2019 | ✓ | ✓ | - | ✓ | - | -
Li et al. [29] | 2020 | ✓ | ✓ | - | ✓ | - | -
Tahir and Alexander [39] | 2022 | ✓ | - | - | ✓ | - | -
Hermann et al. [23] | 2022 | - | ✓ | ✓ | ✓ | - | -
ASAM [3] | 2022 | - | - | - | - | - | -
Proposed Ontology |  | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

Table 1: Comparison of related ontologies and our proposed ontology for
scenario descriptions in the field of autonomous driving.

Next to ontologies which are explicitly designed to describe scenarios, more
exist which also focus on decision-making aspects. In this category, Hummel
[25] developed an ontology capable of describing intersections to a degree,
where the ontology can also be used to infer knowledge about the scenes. While
this is a general attribute of ontologies, she provides a set of rules for the
analysis. Hülsen et al. [26] also describe intersections based on an ontology,
focusing on the road layout, while interactions between entities cannot be
modeled in detail. In [1], this issue is addressed, as Armand et al. focus on
such interactions. They also propose rules to infer knowledge from their
ontology. These rules are partly attributed to the decision-making of an ego
vehicle, e.g., whether it should stop or continue. Due to their strong focus
on actions and interactions, they struggle to describe complex scenarios in a
more general way. Zhao et al. [47] developed a set of three ontologies, namely
Map, Car, and Control. Based on these, they are capable of describing complex
scenes for vehicles only. While the scenes do contain temporal information,
such as paths for vehicles, these are only broad descriptions and not detailed
enough to model complex scenarios. Huang et al. [24] present a similar work
that is able to describe a wide variety of scenarios based on classes for road
networks for highway and urban scenarios, the ego vehicle and its behavior,
static and dynamic objects, as well as scenario types. However, it is designed
to derive driving decisions from the descriptions instead of simulating these
scenarios. Chen and Kloul [13] on the other hand propose an ontology that is
primarily designed to describe highway scenarios, with a special focus on
weather circumstances.

To model corner case scenarios, the requirements for an ontology are very
complex. In general, it needs to be able to describe all types of scenes and
scenarios. For the temporal context, an ontology needs to be able to a)
describe scenarios. Furthermore, it needs to be able to b) describe arbitrary
environments and c) arbitrary objects. Following an open world assumption, we
define ”arbitrary”, in respect to environments and objects, as the possibility
to include such without changing any classes or properties of the ontology.
This means, e.g., referencing to external sources, such as OpenDRIVE files for
environments or CAD files for objects. An ontology needs to be designed in a
way that d) the described scenarios can also be simulated. Finally, e)
information about the corner case levels needs to be included for details and
knowledge extraction. In Table 1, we provide an overview of the previously
introduced ontologies related to these attributes and also mention, whether
the ontology itself is published online. While some authors, such as [16, 25],
released their ontologies previously, the provided links do not contain them
anymore, which is why we excluded outdated sources. A trend can be observed,
where recent approaches focus more on the aspect of scenario simulation.
However, to the best of our knowledge, there exists no ontology to date that
is able to describe and simulate long-tail corner case events. Our proposed
ontology fills this gap, being able to generate ontology scenarios for all
corner case levels and execute them in simulation.

##  3 Method

In order to generate corner case scenarios, we have developed a Master
Ontology which is the foundation for the creation of specific scenarios and
provides the structure for all elements of a scenario. Based on this, all
common corner case categories found in the literature can be addressed. For
the creation of scenarios, we have developed an Ontology Generator module,
which is our interface to human Scenario Designers which do not need any
expertise in the field on ontologies in order to design scenarios. For each
designed scenario, a concrete Scenario Ontology is created. This is a major
advantage over purely coded scenarios, as the complete scenario description is
available in a human- and machine-readable form, which directly enables
knowledge extraction, analysis, and further processing, such as exports into
others formats or combinations of scenarios, for all created scenarios on any
level of detail. Finally, our OpenSCENARIO Conversion module converts this
ontology into an OpenSCENARIO file, which can directly be simulated in the
CARLA simulator. An overview can be found in Fig. 1.

Corner Case Taxonomy OpenSCENARIO Language Master Ontology Scenario Designer
Scenario Ontology Ontology Generator OpenSCENARIO Conversion Execution in
Simulation $1:n$

Figure 1: Flow diagram of our proposed method for the description and
generation of corner case scenarios. Based on a corner case taxonomy and the
OpenSCENARIO language, a Master Ontology was developed, containing all
necessary attributes to describe complex scenarios. In a $1:n$ relation,
ontologies describing individual scenarios can be derived. In an automated
fashion, these scenarios are then converted into the OpenSCENARIO format,
enabling the direct execution in simulation environments.

###  3.1 Master Ontology

At first, we describe the Master Ontology, which is the skeleton of every
concrete scenario. With its help, different scenarios can be described by
instantiating the different classes, using individuals, and setting property
assertions between them. The Master Ontology is closely aligned to the
OpenSCENARIO documentation [4], since the ontology is used for automatic
generation of scenarios. Within the ontology, it is also possible to describe
concrete categories of a corner cases, based on the categories introduced by
Breitenstein et al. [9].

Figure 2: Master ontology, best viewed at 1,600 % and in color. The ontology
is capable to describe scenarios based on the seven sections scenario and
environment, entities, main scenario elements, actions, conditions, weather
and time of day, and corner case level. These seven sections are further
explained in Sec. 3.1. Adapted from [20].

The master ontology, as shown in Fig. 2, consists of 100 classes, 53 object
properties, 44 data properties, 67 individuals, and 683 axioms. The 100
classes are either classes for the description of the corner case category or
derived from the OpenSCENARIO documentation [4], which means that the
definitions of the different OpenSCENARIO elements can also be found there.
They are used as parents for the different individuals we have created within
the ontology. The 53 object properties and the 44 data properties are used to
connect the different parts of a scenario, in order to embed individuals into
concrete scenarios. For a better understanding and more structured
explanation, the proposed Master Ontology can be divided into seven main
groups - Scenario and Environment, Entities, Main Scenario Elements, Actions,
Conditions, Weather and Time, and Corner Case Level. We will describe these in
more detail in the following, with each section marked with an individual
color in Fig. 2.

Scenario and Environment (red). In order to be able to describe a scenario,
the Master Ontology provides the Scenario class, which acts as the root of the
ontology. Together with the scenario class, different object and data
properties are provided. Those are used as connections between the different
scenario elements, such as the Entities, Towns or the Storyboard. Towns are
CARLA specific environments used in the ontology. CARLA allows users to create
custom and thus arbitrary environments.

Entities (green). This group holdes the different entities Vehicle,
Pedestrian, Bicycle, and Misc. For arbitrary Entities, the Misc class can be
utilized. If specific movement patterns are wanted, the classes Vehicle,
Pedestrian and Bicycle are also already available. The individuals can be then
connected to 3D assets from the CARLA blueprint library [10], which can be
extended with external objects. This way, a Scenario Designer is able to add
arbitrary assets into a scenario.

Main Scenario Elements (yellow). The main scenario elements are used to build
the core of any scenario. The highest level is the Storyboard, which includes
an Init and a Story. A Story has at least one Act, which needs at least a
StartTrigger and can optionally include a StopTrigger. Acts also are a
container for different ManeuverGroups, that logically include Maneuvers. The
Maneuvers then have to have minimum one Event, which is also activated by a
StartTrigger. At last, each Event needs to include at least one Action. These
are the main components of the OpenSCENARIO scenario description language and
thus necessary parts of each scenario. For each of them, also a corresponding
connecting property exists, i.e. has_event, has_action, has_init_action.

Actions (dark blue). To be able to describe the maneuvers of the different
Entities, different Actions are represented within the Ontology. Those
include, e.g., TeleportAction, which sets the position of an Entity, or
RelativeLaneChangeAction, which describes a lane change of an Entity.

Conditions (light blue). As part of the StartTrigger and StopTrigger elements,
Conditions are used to activate them. Conditions are divided into the two
subclasses ByEntityCondition and ByValueCondition. In general the difference
between those two is that the ByEntityCondition is always in regard to an
entity, i.e., how close a vehicle is to another vehicle, while the
ByValueCondition is always in regard to a value, i.e., the passed simulation
time. Depending on the type of the Condition, different values must be met in
order for the StartTrigger or StopTrigger to get activated. As an example, the
SimulationTimeCondition can be used as a trigger with respect to the
simulation time, using arithmetic rules.

Weather and Time of Day (orange). To set the weather, the underlying CARLA
town can be modified individually. This includes the weather conditions, which
are subdivided into fog, precipitation, and the position of the sun. Also, the
time of day can be set.

Corner Case Level (pink). In the long tail of rare scenarios, each can be
related to a specific corner case level. The first ones to propose an
extensive taxonomy on these were Breitenstein et al. [7] with a focus on
camera-based sensor setups. This taxonomy was extended by Heidecker et al.
[22] to generalize it to a set of three top-level layers, namely Sensor Layer,
Content Layer, and Temporal Layer, as shown in Fig. 3. In this work, we focus
on camera-related corner cases, which is why the master ontology uses a mixed
model, where the top level layers from Heidecker et al. and the underlying
corner case levels from Breitenstein et al. are used, as shown in Fig. 2,
making a future extension of the master ontology to further sensors
effortless, as they fall into the same top level layers. Occurrences on the
hardware or physical-level, such as dead pixels or overexposure, can be
simulated with subsequent scripts during the simulation phase. Details on
those corner cases can be placed in the individual scenario ontologies by
creating specific individuals of the respective corner case classes of the
Master Ontology.

Figure 3: Corner Case Categorization from Heidecker et al. [22]. The columns
show different layers and levels of corner cases, while the rows are related
to specific sensors. For each combination, examples are provided.

Next to those groups, additional 67 individuals exist, which are divided into
Constants and Default Individuals. There are two types of constants:
OpenSCENARIO Constants, such as arithmetic or priority rules, and CARLA
Constants, such as assets. The default individuals are used to help a Scenario
Designer to create scenarios faster and easier. These include common patterns,
such as default weather conditions or a trigger, which activates when the
simulation starts running. In addition, a default ego vehicle is also included
in the Master Ontology, which has a set of cameras and a BoundingBox attached
to it. As the last part of the ontology, the 683 axioms represent the
connections and rules between the entities and the properties within the
ontology, along with the individuals.

###  3.2 Scenario Ontology Generation

The manual creation of ontologies is a very time-consuming, exhausting and
error-prone process, which additionally needs expertise in the general field
of ontologies and related software. Thus, and to ensure, that the OpenSCENARIO
Conversion module functions properly, we have developed the Ontology Generator
module, which takes as input a scripted version of a scenario and creates a
Scenario Ontology as a result. The concept behind the Ontology Generator is to
use the Master Ontology as a base for a scenario description and automatically
create the necessary individuals and property assertions between them. To read
and write ontologies, we utilize the python library Owlready2 [34]. The Master
Ontology is read by the Ontology Generator, and it uses, depending on the
scenario, all classes, properties, and default individuals needed. The result
is a new Scenario Ontology, which has the same structure as the Master
Ontology with respect to classes and properties, but includes newly created
individuals for the scenario designed by the Scenario Designer.

Main Scenario Elements | Entities | Actions | Environment and Weather
---|---|---|---
newScenario | newEgoVehicle | newEnvironmentAction | newEnvironment
newStoryboard | newCar | newSpeedAction | newTimeOfDay
newInit | newPedestrian | newTeleportActionWithPosition | newWeather
newStory | newMisc | newTeleportActionWithRPAO | newFog
newAct |  | newRelativeLaneChangeAction | newPrecipitation
newManeuverGroup |  |  | newSun
newManeuver |  |  | changeWeather
newEvent |  |  |
newAction |  |  |
Conditions | Assets | Other |
newSimulationCondition | newAsset | newStopTrigger |
newRelativeDistanceCondition | getBicycleAssets | newStartTrigger |
newStoryboardESCondition | getCarAssets | newRoadCondition |
newTraveledDistanceCondition | getPedestrianAssets | newTransitionDynamics |
| getMiscAssets | setCornerCase |

Table 2: Overview of methods of the Ontology Generator module, which are
available to a human Scenario Designer. The methods are divided in seven
groups, five from which were introduced in Sec. 3.1. The groups Assets and
Other are related to 3D objects and miscellaneous functions, respectively.

1 $import~{}OntologyGenerator~{}as~{}OG$

2$import~{}MasterOntology~{}as~{}MO$

$ego\\_vehicle\leftarrow MO.ego\\_vehicle\;$ //Default ego vehicle individual

3

$weather\\_def\leftarrow MO.def\\_weather\;$ //Default weather individual

4

5$Initialize~{}teleport\\_action(ego\\_vehicle),speed\\_action(ego\\_vehicle)$

6$init\\_scenario\leftarrow
OG.newInit(speed\\_action,teleport\\_action,weather\\_def)$

//Starting conditions for storyboard

7

8$Initialize~{}traveled\\_distance\\_condition$

9$Trigger\leftarrow OG.newStartTrigger(traveled\\_distance\\_condition)$

//Trigger Condition: Ego vehicle travelled defined distance

10

11$Initialize~{}weather(sun,fog,precipitation)$

12$Initialize~{}time\\_of\\_day,road\\_condition$

13$env\leftarrow OG.newEnv(time\\_of\\_day,weather,road\\_condition)$

14$env\\_action\leftarrow OG.newEnvAction(env)$

//Foggy environment after trigger

15

16$Initialize~{}Event,Maneuver,ManeuverGroup,Act,Story,Storyboard$

//Necessary OpenSCENARIO elements

$Export~{}ScenarioOntology$

Algorithm 1 Creation of a scenario ontology with the Ontology Generator, where
the ego vehicle enters a foggy area (incl. abstract elements)

Since the Master Ontology is built based on the OpenSCENARIO documentation
[2], which is a very powerful and flexible framework, it allows for many
possible combinations. This gives a Scenario Designer a large flexibility with
respect to the design of new scenarios. The Ontology Generator is well
documented. This way, no prior experience with the OpenSCENARIO format is
necessary. Table 2 provides an overview of the functions available to the
human Scenario Designer to create scenarios.

With the help of the Ontology Generator, every part which was defined within
the Master Ontology can be utilized. For example, the functions within the
first group shown in Table 2, are used to create every scenario main element.
Algorithm 1 shows, how a partly abstracted implementation, as done by a
Scenario Designer, looks like. The full example can be found in the GitHub
repository. In Sec. 4 we demonstrate an exemplary Scenario Ontology which was
generated by the Ontology Generator. In this demonstration, the scenario
ontology from Algorithm 1 is related to the visualization in Fig. 4.

###  3.3 Scenario Simulation

After a scenario is described with the help of individuals within a scenario
ontology, it is being read by the OpenSCENARIO Conversion module, as shown in
Fig. 1. From these concrete scenarios, the conversion module generates
OpenSCENARIO files. These can can be directly simulated without any further
adjustments. Since the OpenSCENARIO files include simulator-specific details,
we have focused on the CARLA222CARLA version 0.9.13 was utilized simulation
environment [11]. In an earlier stage, we were also able to show compatibility
with the esmini [15] environment.

When the Ontology Generator module is used to create the scenario ontologies,
their structural integrity is ensured, which is a necessary requirement for
the conversion module. This means that each scenario ontology is correctly
provided to be processed by the conversion module. Theoretically, scenario
ontologies could also be created manually to be processed by the conversion
module. However, human errors are likely, preventing the correct processing by
the conversion module.

While each scenario ontology is able to cover multiple corner cases, the
created ontologies are fully modular. This means, given the same environment,
our method is capable of combining multiple, already existing scenario
ontologies into a new single scenario ontology. In such cases, where the
number of scenario individuals is $n>1$, a pre-processing stage is triggered
which extends the ontology to combine all $n$ provided scenarios into a single
new scenario $S_{fusion}$. For this purpose, this stage creates a new
scenario, storyboard, and init. Subsequently, for every included scenario, the
algorithm goes through its stories, entities, and init actions and merges them
in $S_{fusion}$. For the final creation of the OpenSCENARIO file, the
conversion module utilizes the property assertions between individuals to
create the according python objects, which are then used by the PYOSCX library
[36] to create the OpenSCENARIO file. These files can then be read by the
ScenarioRunner [12] and executed in CARLA. In the following Sec. 4, we
demonstrate a set of ten simulated scenarios.

##  4 Evaluation

For the evaluation, we have created a diverse scenario catalog containing
scenarios from all corner case levels. Following Breitenstein et al. [8],
these cover different levels of complexity and thus criticalities, starting
with simpler sensor layer cases and ending with highly complex temporal layer
corner cases. For the qualitative evaluation we show the feasibility of the
approach as shown in Fig. 1 and demonstrate it with a set of ten scenarios,
that descriptions made by a human Scenario Designer get translated into proper
Scenario Ontologies and correctly simulated.

# | Corner Case Level | Individuals | Scenario Description
---|---|---|---
(a) | Sensor Layer - Hardware Level | - | Dead Pixel: Camera sensor affected
(b) | Content Layer - Domain Level | 94 | Domain Shift: Sudden weather change
(c) | Content Layer - Object Level | 93 | Single-Point Anomaly: Unknown object on the road
(d) | Content Layer - Scene Level | 164 | Collective Anomaly: Multiple known objects on the road
(e) | Content Layer - Scene Level | 111 | Contextual Anomaly: Known non-road object on the road
(f) | Temporal Layer - Scenario Level | 94 | Novel Scenario: Unexpected event in another lane
(g) | Temporal Layer - Scenario Level | 104 | Risky Scenario: A risky maneuver
(h) | Temporal Layer - Scenario Level | 95 | Anomalous Scenario: Unexpected traffic participant behaviour
(i) | Combined: (d) and (f) | 156 | Combined: Collective and Novel Scenario
(j) | Combined: (f) and (h) | 122 | Combined: Novel and Anomalous Scenario

Table 3: Overview of scenario ontologies, which were derived from the master
ontology and subsequently executed in simulation. These exemplary scenarios
cover all corner case categories.

For the selection of the exemplary corner case scenarios, we considered three
types of sources. First, we used examples provided by the literature, such as
the ones provided by Breitenstein et al. [9]. Second, various video sources,
such as third-person videos, and dash-cam videos of traffic situations [31]
were used for inspiration. Third, multiple brainstorming sessions took place,
where personal experiences were collected. Afterwards, we narrowed the
selection down to a set of eight representative scenarios. Two more were
created by combining two of those eight scenarios. An overview of these
scenarios can be found in Table 3. Visualizations of all scenarios can be
found in Fig. LABEL:fig:examples.

In the a) Dead Pixel scenario, an arbitrary scenario can be chosen, where only
the sensor itself will be affected. The b) Domain Shift is a sudden weather
change, where we created a scenario where the ego vehicle suddenly drives into
a dense fog. For the c) Single-Point Anomaly, we chose to simulate a falling
vending machine on the road. This scenario also inspired the next scenario, d)
Contextual Anomaly, which also has falling objects on the road, but in this
case the objects are traffic signs. This can be considered, for example, in a
very windy environment. As a e) Collective Anomaly, we chose to simulate a lot
of running pedestrians in front of the ego vehicle, which can for example
happen during a sports event. For the f) Novel Scenario, we described a
scenario where a cyclist performs unexpected maneuvers in the opposite lane.
The g) Risky Scenario which we chose is a close cut-in maneuver in front of
the ego vehicle. The last corner case category is the h) Anomalous scenario,
for which we have chosen a pedestrian who suddenly runs in front of the ego
vehicle. To demonstrate the scalability of our approach, we have also combined
scenarios. In the i) combination of Novel Scenario and the Collective
Scenario, a lot of running pedestrians are in front of the ego vehicle, while
a cyclist performs unexpected maneuvers in the opposite lane, next to the
pedestrians. In addition, we also merged the Novel Scenario with the Anomalous
Scenario, resulting in j) a pedestrian walking in front of the ego vehicle and
the cyclist.

###  4.1 Scenario Ontologies

At the core of each demonstrated scenario lies a Scenario Ontology. In the
following, we present the construction of an exemplary scenario and describe
how it is represented in the Scenario Ontology with individuals. An example
graph of the exemplary ontology can be seen in Fig. 4. The ontology has 94
individuals, which means that 27 new individuals were created, since the
Master Ontology has 67 default individuals.

In Fig. 4, every used class, property and individual is presented. Each
individual, which name starts with ”indiv_”, is a newly created part of the
Scenario ontology, every other individual is either a default or a constant.
The graph starts from the top with the Scenario individual, which is connected
to a CARLA town and a newly created Storyboard. As mentioned earlier, every
Storyboard has an Init and a Story. In this particular Init, there are only
the Actions, which are responsible for the position and the speed of the
EgoVehicle, and connections to the default EnvironmentAction. The most
interesting part of this Scenario however can be found deep within the Story
\- namely the second EnvironmentAction, which creates a dense fog inside the
scenario. This Action gets triggered by the indiv_DistanceStartTrigger, which
has a TraveledDistanceCondition as a Condition. Since this type of Condition
is an EntityCondition, it requires a connection to an Entity, in our case the
ego_vehicle. This StartTrigger gets activated when the ego_vehicle has
travelled a certain distance. After this Event is executed, the Scenario comes
to its end.

Descriptions and visualizations of the remaining nine Scenario Ontologies can
be found in [20], and the code for all ten scenarios, in order to recreate
them, can be found in the GitHub repository.

Figure 4: Scenario ontology describing a vehicle entering a foggy area. Best
viewed at 1,600 %. The ontology describes the scenario based on the seven
sections scenario and environment, entities, main scenario elements, actions,
conditions, weather and time of day, and corner case level. Reprinted from
[20].

##  5 Conclusion

Our work focuses on the generation of rare corner case scenarios in order to
provide additional input training data as well as test and evaluation
scenarios. This way, it contributes to potentially safer deep neural networks,
which might be able to become more robust to anomalies. The proposed Master
Ontology is the core of our approach and enables the creation of specific
scenarios for all of the corner case levels developed by Breitenstein et al.
[9]. From the one Master Ontology, concrete scenario ontologies can be
derived. The whole process after the design of such an ontology is fully
automated, which includes the generation of ontologies itself, based on the
human input for the Ontology Generator as well as the conversion into the
OpenSCENARIO format. This allows for a direct simulation of the scenarios in
the CARLA simulation environment without any further adjustments. Since
ontologies are a highly complex domain, we have put an emphasis on the design
of the Ontology Generator module, which does not require expertise in the
field of ontologies for a human to create scenarios. We have demonstrated our
approach with a set of ten concrete scenarios, which cover all corner case
levels.

As an outlook, we would first like to discuss existing limitations of our
work. At the moment, the master ontology is focused on camera-related corner
cases, which is why the current implementation only includes hardware-related
corner cases for camera sensors. Also, the subsequent scripts for camera-
related corner cases are currently not triggered automatically, but manually.
For the Master Ontology, we have implemented a vast body of the OpenSCENARIO
standard in order to demonstrate our designed scenarios. However, for future
scenarios, modifications of the master ontology might be necessary. We have
provided instructions for such extensions in our GitHub repository. While the
master ontology supports arbitrary objects, in our demonstrated scenarios we
were only able to utilize assets which were already included within CARLA, as
we ran into compilation issues with respect to the Unreal Engine and the
utilized CARLA version. Our learnings as well as ideas to address this issue
can also be found in the GitHub repository.

Finally, we would like to point out future directions. The extraction of
corner case levels from the ontology itself could be automated, e.g., with
knowledge extraction methods based on Semantic Web Rule Language (SWRL) rules.
However, this is a challenging field itself. Based on the generated scenarios,
which were created by human Scenario Designers, automated variations can be
introduced to drastically increase the number of available scenarios, as shown
by [29, 23, 30]. This way, a powerful combination of knowledge- and data-
driven scenario generation can be achieved for the long tail of rare corner
cases.

## Acknowledgment

This work results partly from the project KI Data Tooling (19A20001J) funded
by the Federal Ministry for Economic Affairs and Climate Action (BMWK).

## References

  * [1] Armand, A., Filliat, D., Ibañez-Guzman, J.: Ontology-based context awareness for driving assistance systems. In: IEEE Intelligent Vehicles Symposium Proceedings (2014)
  * [2] ASAM: ASAM OpenSCENARIO. https://www.asam.net/standards/detail/openscenario, accessed: 2022-02-28
  * [3] ASAM: ASAM OpenXOntology. https://www.asam.net/project-detail/asam-openxontology/, accessed: 2022-02-28
  * [4] ASAM: OpenSCENARIO Documentation. https://releases.asam.net/OpenSCENARIO/1.0.0/ASAM_OpenSCENARIO_BS-1-2_User-Guide_V1-0-0.html, accessed: 2022-01-28
  * [5] Bagschik, G., Menzel, T., Maurer, M.: Ontology based scene creation for the development of automated vehicles. In: IEEE Intelligent Vehicles Symposium (IV) (2018)
  * [6] Bogdoll, D., Breitenstein, J., Heidecker, F., Bieshaar, M., Sick, B., Fingscheidt, T., Zöllner, M.: Description of Corner Cases in Automated Driving: Goals and Challenges. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2021)
  * [7] Breitenstein, J., Termöhlen, J.A., Lipinski, D., Fingscheidt, T.: Systematization of corner cases for visual perception in automated driving. In: IEEE Intelligent Vehicles Symposium (IV) (2020)
  * [8] Breitenstein, J., Termöhlen, J.A., Lipinski, D., Fingscheidt, T.: Systematization of corner cases for visual perception in automated driving. In: IEEE Intelligent Vehicles Symposium (IV) (2020)
  * [9] Breitenstein, J., Termöhlen, J.A., Lipinski, D., Fingscheidt, T.: Corner cases for visual perception in automated driving: Some guidance on detection approaches. arXiv:2102.05897 (2021)
  * [10] CARLA: CARLA Blueprint Library. https://carla.readthedocs.io/en/latest/bp_library/, accessed: 2022-02-28
  * [11] CARLA: CARLA Simulator. https://carla.org/, accessed: 2022-02-28
  * [12] CARLA: Scenario Runner Github. https://github.com/carla-simulator/scenario_runner, accessed: 2022-02-28
  * [13] Chen, W., Kloul, L.: An ontology-based approach to generate the advanced driver assistance use cases of highway traffic. In: Proceedings of the 10th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (KEOD) (2018)
  * [14] Ding, W., Chen, B., Li, B., Eun, K.J., Zhao, D.: Multimodal Safety-Critical Scenarios Generation for Decision-Making Algorithms Evaluation. IEEE Robotics and Automation Letters 6 (2021)
  * [15] esmini: Esmini github. https://github.com/esmini/esmini, accessed: 2022-02-28
  * [16] Fuchs, S., Rass, S., Lamprecht, B., Kyamakya, K.: A model for ontology-based scene description for context-aware driver assistance systems. In: International ICST Conference on Ambient Media and Systems (2010)
  * [17] de Gelder, E., Paardekooper, J.P., Saberi, A.K., Elrofai, H., Camp, O.O.d., Kraines, S., Ploeg, J., De Schutter, B.: Towards an ontology for scenario definition for the assessment of automated vehicles: An object-oriented framework. IEEE Transactions on Intelligent Vehicles (2022)
  * [18] (Germany), B.: Verordnung zur Regelung des Betriebs von Kraftfahrzeugen mit automatisierter und autonomer Fahrfunktion und zur Änderung straßenverkehrsrechtlicher Vorschriften. https://dserver.bundestag.de/brd/2022/0086-22.pdf (2022), accessed: 2022-06-15
  * [19] (Germany), B.: Entwurf eines Gesetzes zur Änderung des Straßenverkehrsgesetzes und des Pflichtversicherungsgesetzes – Gesetz zum autonomen Fahren. https://www.bmvi.de/SharedDocs/DE/Anlage/Gesetze/Gesetze-19/gesetz-aenderung-strassenverkehrsgesetz-pflichtversicherungsgesetz-autonomes-fahren.pdf?__blob=publicationFile (2021), accessed: 2022-06-15
  * [20] Guneshka, S.: Ontology-based corner case scenario simulation for autonomous driving. Bachelor thesis, Karlsruhe Institute of Technology (KIT) (2022)
  * [21] Hanselmann, N., Renz, K., Chitta, K., Bhattacharyya, A., Geiger, A.: King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients. arXiv:2204.13683 (2022)
  * [22] Heidecker, F., Breitenstein, J., Rösch, K., Löhdefink, J., Bieshaar, M., Stiller, C., Fingscheidt, T., Sick, B.: An application-driven conceptualization of corner cases for perception in highly automated driving. In: IEEE Intelligent Vehicles Symposium (IV) (2021)
  * [23] Herrmann, M., Witt, C., Lake, L., Guneshka, S., Heinzemann, C., Bonarens, F., Feifel, P., Funke, S.: Using ontologies for dataset engineering in automotive ai applications. In: Design, Automation and Test in Europe Conference and Exhibition (DATE) (2022)
  * [24] Huang, L., Liang, H., Yu, B., Li, B., Zhu, H.: Ontology-based driving scene modeling, situation assessment and decision making for autonomous vehicles. In: Asia-Pacific Conference on Intelligent Robot Systems (ACIRS) (2019)
  * [25] Hummel, B.: Description Logic for Scene Understanding at the Example of Urban Road Intersections. Ph.D. thesis, Karlsruhe Institute of Technology (KIT) (2009)
  * [26] Hülsen, M., Zöllner, J.M., Weiss, C.: Traffic intersection situation description ontology for advanced driver assistance. In: IEEE Intelligent Vehicles Symposium (IV) (2011)
  * [27] Karpathy, A.: Tesla Autonomoy Day. https://youtu.be/Ucp0TTmvqOE?t=8671 (2019), accessed: 2022-06-15
  * [28] Klueck, F., Li, Y., Nica, M., Tao, J., Wotawa, F.: Using ontologies for test suites generation for automated and autonomous driving functions. In: IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) (2018)
  * [29] Li, Y., Tao, J., Wotawa, F.: Ontology-based test generation for automated and autonomous driving functions. Information and Software Technology 117 (2020)
  * [30] Menzel, T., Bagschik, G., Isensee, L., Schomburg, A., Maurer, M.: From functional to logical scenarios: Detailing a keyword-based scenario description for execution in a simulation environment. In: IEEE Intelligent Vehicles Symposium (IV) (2019)
  * [31] Minute, M.: Trees falling on road. https://www.youtube.com/watch?v=3VsLeUtXvxk&ab_channel=Mad1Minute (2017), accessed: 2022-07-21
  * [32] Noy, N., Mcguinness, D.: Ontology development 101: A guide to creating your first ontology. Knowledge Systems Laboratory 32 (2001)
  * [33] On-Road Automated Driving Committee: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles. Standard J3016-202104, SAE International (2021)
  * [34] OWLReady2: Welcome to Owlready2’s documentation! https://owlready2.readthedocs.io/en/v0.36/ (2021), accessed: 2022-02-28
  * [35] Pretschner, A., Hauer, F., Schmidt, T.: Tests für automatisierte und autonome Fahrsysteme. Informatik Spektrum 44 (2021)
  * [36] pyoscx: scenariogeneration. https://github.com/pyoscx/scenariogeneration (2022), accessed: 2022-07-20
  * [37] Rempe, D., Philion, J., Guibas, L.J., Fidler, S., Litany, O.: Generating useful accident-prone driving scenarios via a learned traffic prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [38] Schoener, H.P., Mazzega, J.: Introduction to pegasus. In: China Autonomous Driving Testing Technology Innovation Conference (2018)
  * [39] Tahir, Z., Alexander, R.: Intersection focused situation coverage-based verification and validation framework for autonomous vehicles implemented in carla. In: Mazal, J., Fagiolini, A., Vasik, P., Turi, M., Bruzzone, A., Pickl, S., Neumann, V., Stodola, P. (eds.) Modelling and Simulation for Autonomous Systems (2022)
  * [40] Tao, J., Li, Y., Wotawa, F., Felbinger, H., Nica, M.: On the industrial application of combinatorial testing for autonomous driving functions. In: IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (2019)
  * [41] Tuncali, C.E., Fainekos, G., Prokhorov, D., Ito, H., Kapinski, J.: Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Transactions on Intelligent Vehicles (2020)
  * [42] Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., Maurer, M.: Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving. In: Proc. of ITSC (2015)
  * [43] Wang, J., Pun, A., Tu, J., Manivasagam, S., Sadat, A., Casas, S., Ren, M., Urtasun, R.: AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
  * [44] Waymo: Waymo One. https://waymo.com/waymo-one/ (2022), accessed: 2022-06-15
  * [45] Wotawa, F., Li, Y.: From ontologies to input models for combinatorial testing. In: International Conference on Testing Software and Systems (ICTSS) (2018)
  * [46] Zaid, T.: Intersection focused Situation Coverage-based Verification and Validation Framework for Autonomous Vehicles Implemented in CARLA. https://github.com/zaidtahirbutt/Situation-Coverage-based-AV-Testing-Framework-in-CARLA (2022), accessed: 2022-07-20
  * [47] Zhao, L., Ichise, R., Liu, Z., Mita, S., Sasaki, Y.: Ontology-based driving decision making: A feasibility study at uncontrolled intersections. IEICE Transactions on Information and Systems (2017)