#  Open-Source Tool Based Framework for Automated Performance Evaluation of an
AD Function
††thanks: *These authors contributed equally to this work.

1st Sanath Konthala  Institute Thermodynamic/Combustion Technology
FH Aachen - University of Applied Sciences
Aachen, Germany
<EMAIL_ADDRESS>2nd Daniel Becker  Institute for
Automotive Engineering
RWTH Aachen University
Aachen, Germany
<EMAIL_ADDRESS>3rd Lutz Eckstein  Institute for Automotive
Engineering
RWTH Aachen University
Aachen, Germany
<EMAIL_ADDRESS>
###### Abstract

As automation in the field of automated driving (AD) progresses, ensuring the
safety and functionality of AD functions (ADFs) becomes crucial. Virtual
scenario-based testing has emerged as a prevalent method for evaluating these
systems, allowing for a wider range of testing environments and
reproducibility of results. This approach involves AD-equipped test vehicles
operating within predefined scenarios to achieve specific driving objectives.
To comprehensively assess the impact of road network properties on the
performance of an ADF, varying parameters such as intersection angle,
curvature and lane width is essential. However, covering all potential
scenarios is impractical, necessitating the identification of feasible
parameter ranges and automated generation of corresponding road networks for
simulation. Automating the workflow of road network generation, parameter
variation, simulation, and evaluation leads to a comprehensive understanding
of an ADF’s behavior in diverse road network conditions. This paper aims to
investigate the influence of road network parameters on the performance of a
prototypical ADF through virtual scenario-based testing, ultimately advocating
the importance of road topology in assuring safety and reliability of ADFs.

###### Index Terms:

automated driving, scenario-based testing, safety, simulation, open-source

##  I Introduction

The rise of automated driving (AD) has marked a significant milestone in the
evolution of the automotive industry, revolutionizing the concept of
automobiles. The advent of driverless cars has ushered in a transformative
advancement, empowering vehicles to operate autonomously and reducing reliance
on human drivers. However, amidst the promises of AD, its impact on traffic
safety remains the most crucial aspect. The assertion of accident reduction
and improved safety presents a profound opportunity and a daunting challenge
for ADFs. Ensuring that these systems surpass human drivers is a complex task.
Experts agree that the conventional approach of conducting statistical tests,
as done in the automotive industry for previous generations, is not feasible
[1]. Some sources advocate the need for testing over billions of kilometers
[2], as per International Organization for Standardization (ISO) norm 26262
[3], throughout the development lifecycle.

Figure 1: Overview of the implemented automated scenario-based testing
methodology. All components and formats are open-source.

To address these challenges for assessment of ADFs, virtual scenario-based
testing is a promising solution [4]. In this approach, an ADF is challenged by
creating predefined scenarios that simulate real-world driving conditions. To
avoid arbitrary specifications of scenarios, the New Assessment/Test Method
(NATM) guidelines propose a structured approach for classifying and describing
scenarios based on different levels of abstraction [5]. At the highest level,
functional scenarios provide a fundamental description of the scenario,
including the ego vehicle’s actions, interactions with other road users and
objects, road geometry and environmental conditions. Logical scenarios define
value ranges or probability distributions for each element, allowing for a
range of possible scenarios. At the most explicit level, concrete scenarios
are developed by selecting specific values for each element, ensuring a
diverse set of test situations and reproducibility. In this work, we perform
scenario-based testing in an attempt to understand the influence of geometric
parameters alone on an ADF’s performance. The overall framework is sketched in
Fig. 1.

The remainder of this paper is structured as follows. First, an overview on
related work is given in Sec. II, followed by an introduction of the open-
source tools we utilized in our implementation (cf. Sec. III). In Sec. IV, the
overall methodology is described in which we first define logical scenarios by
specifying value ranges for the varied parameters. We then generate sets of
concrete scenarios. Subsequently, the ADF is tested on each scenario, and its
performance is evaluated using a set of Key Performance Indicators (KPIs).
After that, some results are discussed in Sec. V and finally, we draw a
conclusion and give an outlook (cf. Sec. VI).

##  II Related Work

This chapter provides an overview of the existing research related to the
assessment of ADFs and the underlying motivation behind this study.

###  II-A Scenario-based Testing

Virtual scenario-based testing offers a promising alternative to real-world
testing for ADFs, enabling the evaluation of driving functions under diverse
situations by varying scenario parameters. However, a significant challenge,
known as “parameter space explosion” arises due to the large number of
parameters defining a scenario, leading to an exponentially increasing number
of possible parameter combinations [6]. As a result, it becomes impractical to
include all theoretically possible scenarios in the safety validation process.
To address this, [6] proposes a method to define the required test coverage
for scenario-based validation. In [7], a comprehensive literature review on
scenario-based testing for ADF is conducted. They explored various approaches
for defining and selecting scenarios to achieve efficient testing and
conducted a comparative analysis of different methods. According to the
taxonomy of the scenario-based testing introduced by them, the source of the
scenarios is from abstract knowledge of experts, like German guidelines for
the construction of highways [8] or from existing driving and accident data.

###  II-B Scenario Contents

The scenario description in scenario-based testing is critical to ensure
comprehensive testing and proper execution. The 6 layer model (6LM) shown in
Fig. 2 offers a structured approach with six layers for organizing essential
elements of a scenario [9]. Layer 1 focuses on permanent objects required for
traffic guidance, such as road network related elements. Layer 2 includes non-
road objects like buildings and vegetation. Moving to Layer 3, temporary
objects from layers 1 and 2 are described, such as road work signs or fallen
trees. Layer 4 introduces time-dependent description, covering dynamic objects
that move or can potentially move. This layer includes traffic participants,
parked cars, and stationary pedestrians. In Layer 5, environmental conditions
like weather and lighting are accounted for, with effects that may change over
time. Layer 6 incorporates digital data, including information exchange,
communication, and traffic light status. Intelligent traffic management
systems using V2X and V2V communication or cellular networks fall into this
layer.

Figure 2: The 6 layer model to structure a scenario according to [9].

Existing research on safety assurance for ADFs has primarily focused on layers
3, 4, and 5 of the 6LM [7, 10]. However, safety assurance concerning layer 1,
i.e., road network geometry, has been lacking, leaving gap in understanding
the impact of road geometry design on driving function performance and safety.
In [11] an analysis of the relationship between road geometry and automated
driving safety for vehicle-based mobility services is conducted. Their study
was limited to simulations and evaluation based on a real-time dataset,
lacking coverage of a wide range of scenarios. However, the study highlights
the importance of considering road network variations to simulate ADFs and
evaluate their performance, ultimately ensuring safety assurance of the
driving function. Building upon the work in [12] and [13], this paper extends
the approach for performance analysis of ADFs with the use of open-source
tools. The study aims to test the ADF in a more complex and diverse
environment, further exploring its capabilities and behavior under varied road
network conditions.

##  III Research Approach

The core idea of this work is to use open-source tools and formats for
evaluating the ADF’s performance. This section introduces the existing
standards and open-source tools that have been employed to illustrate the
research approach of the study.

###  III-A Road and Scenario Description Standards

In this section, a brief overview of existing road and scenario description
formats is given.

####  III-A1 OpenDRIVE

ASAM OpenDRIVE is an open standard utilized for detailing road networks and
topologies111https://www.asam.net/standards/detail/opendrive/. It employs XML
syntax with the file extension ”.xodr”, providing a standardization foundation
for describing road networks. The OpenDRIVE file stores various data elements
to outline road and lane geometries, road marks, and other objects like
signals. These road networks can be artificially generated or based on actual
data. While the OpenDRIVE format primarily serves simulations within
simulators and allows for rapid map switching, it poses challenges in
interacting with ADFs.

####  III-A2 Lanelet2

The Lanelet2 framework contains a C++ library and a map format written in XML
[14]. It has been gaining popularity for research on ADFs, as it provides
comprehensive functionality and meets the requirements of both driving
simulators and AD tasks. A Lanelet2 map is an extension of OpenStreetMap,
featuring three layers: the physical layer with observable elements, the
relational layer connecting them to lanes and traffic rules, and the
topological layer forming a network of passable regions. The Lanelet2 library
allows seamless interaction with Lanelet2 files, enabling various operations
like processing and manipulation of map data for AD applications.

####  III-A3 OpenSCENARIO

ASAM OpenSCENARIO is a standardized format and methodology for describing
scenarios in driving and traffic simulators to test
ADFs222https://www.asam.net/standards/detail/openscenario/. It enables
comprehensive descriptions of complex maneuvers involving multiple elements
like vehicles and pedestrians. Scenarios can be derived from driver actions or
recorded driving maneuvers, precisely defining the dynamic content of the
simulated world, including traffic participants’ behavior. The format
introduces a hierarchical structure for scenario descriptions, enabling the
creation of storyboards, stories, acts, and maneuvers. While focusing on
defining the dynamic content, it can reference and integrate road network
descriptions from ASAM OpenDRIVE, providing a comprehensive framework for
describing both static and dynamic aspects of simulation applications.

###  III-B Open-Source Tools and Software

This section provides a brief introduction to the essential tools and software
utilized in this study.

####  III-B1 Road Generation Tool

The Road Generation Tool is an open-source tool that simplifies the creation
of OpenDRIVE road networks using XML-based more logical and less redundant
descriptions than OpenDRIVE [15]. Users can specify parameters and values in
an XML template to generate standardized simulation maps in the OpenDRIVE
format. The tool provides command-line functionality for creating various road
geometries, such as junctions and intersections. It also includes a Python
package that allows the incorporation of stochastic variables for generating
multiple variations of the road network.

####  III-B2 CommonRoad Scenario Designer

The CommonRoad Scenario Designer [16] is a versatile toolbox for creating,
manipulating, and converting road maps and scenarios. It supports multiple map
formats, including Lanelet2, OpenDRIVE, OpenStreetMap (OSM), and SUMO. The
toolbox offers a graphical user interface (GUI), command line interface, and
Python APIs to facilitate the creation, editing, and visualization of
CommonRoad maps and scenarios. One of its key features is the ability to
convert maps between different formats, such as OpenDRIVE to Lanelet2 format.

####  III-B3 Robot Operating System (ROS)

ROS is an open-source meta-operating system333https://www.ros.org/ used for
developing and testing robotic systems, including ADF validation. Its benefits
include modularity, seamless communication, visualization and debugging tools,
a vibrant community, and integration with simulation environments. ROS
provides a flexible and collaborative framework for developing, integrating,
and testing components in ADF.

####  III-B4 CARLA Simulator

CARLA is an open-source simulation platform [17] designed for developing,
training, and validating AD systems. It offers open digital assets like urban
layouts, buildings, and vehicles, allowing flexible specification of sensor
suites, environmental conditions, and full control over static and dynamic
actors. The platform generates realistic and high-fidelity virtual
environments, providing users with a space to test and evaluate their
algorithms and systems. CARLA’s extensive API enables users to control the
simulation, access sensor data, and interact with the environment. The
different CARLA modules used in this study are the ROS bridge, ScenarioRunner
and Metrics Module. The former establishes seamless communication between ROS
topics and CARLA, enabling two-way data exchange. This way, an ADF can
interact effectively with the simulation environment. The ScenarioRunner acts
as the scenario engine within the simulation and can process OpenSCENARIO
files and orchestrates the various entities of a given scenario. Finally, the
metrics module is used for calculations and monitoring of parameters after a
scenario is finished in the CARLA simulator. The recorded simulation
information is used as input, and customized metrics can be defined.

##  IV Methodology

To comprehend the influence of the road network geometry on the performance of
the ADF, it is crucial to conduct tests across various scenarios. For this
purpose, an automated framework has been created that facilitates efficient
scenario generation and simulation. The workflow is visualized in Fig. 1 and
consists of three main components: the automated scenario generation of
logical and concrete instances, an automated simulation framework, and the
simulation evaluation. This streamlined process allows to systematically
explore and assess the impact of road network geometry on the performance of
the ADF and will be detailed in the following sections.

###  IV-A Automated Scenario Generation

To assess the ADF’s performance across diverse road geometry parameters, two
basic road topology templates and a complex road network are utilized. The
variation tool within the road generation tool is employed to facilitate the
creation of various iterations. Different scenarios are generated using a
linear distribution approach for the parameter values, allowing the efficient
creation of a range of varied scenarios and evaluation of the ADF’s behavior
under different road geometries.

For each template, logical scenarios are initially defined by incorporating
various variables that defined the road geometry. These variables represent
different aspects of the road design. Each logical scenario consists of a road
network template and an OpenSCENARIO file containing initial configuration
settings for co-simulation, which encompass the ego vehicle’s start and
destination locations. The OpenSCENARIO file also contains essential
simulation details, such as vehicle information, scenario specifics, and
environment data. As most settings in the OpenSCENARIO file are default and
remain unchanged, this logical OpenSCENARIO file is used in the simulation
framework to generate concrete OpenSCENARIO files. In addition, The variation
tool is used to generate concrete OpenDRIVE maps. All templates consist of
roads with two lanes — one for each direction. It is worth noting that
elevation has not been taken into account while creating scenarios for this
particular study.

The simulation platform requires an OpenDRIVE map, while the AD platform
relies on a lanelet2 map. Consequently, each OpenDRIVE map is converted to a
lanelet2 map with help of the CommonRoad scenario designer. Then, both maps
and the OpenSCENARIO file are stored to form a scenario database. To
efficiently handle the large number of generated scenarios, an automated
approach has been implemented for scenario generation and lanelet2 conversion
using Python scripts. This automated workflow streamlines the process of
scenario creation and ensures the availability of the required files for
simulation execution. The different parameters used to define the logical
scenarios are summarized in Tab. I.

Figure 3: Concrete examples of the three utilized road network templates.
Exemplary routes are drawn in each scenario.

The description of the individual templates used in this study are given as
follows and examples for each template are visualized in Fig. 3.

Table I: Parameter spaces and combinations for the road network generation resulting in a total of 8 logical scenarios. Scenario | Varied parameter | Constant param.
---|---|---
Curved road (left) | Lane width: $3.4$ \- $4.5\,\text{m}$ | Curve radius
Curved road (right) | Lane width: $3.0$ \- $4.0\,\text{m}$ | Curve radius
Curved road (l+r) | Lane width: $50$ \- $500\,\text{m}$ | Lane width
T-junction | Junction gap: $5$ \- $30\,\text{m}$ | Intersection angle
Intersec. angle: $35$ \- $135^{\circ}$ | Junction gap
Lane width: $3.5$ \- $4.2\,\text{m}$ | Intersection angle
Complex road | Lane width: $3.5$ \- $4.0\,\text{m}$ | curve radius

####  IV-A1 Curved Road

The first template employed in this research is a curved road, which serves as
a fundamental exploration of road network parameters like lane width and curve
radius. To examine the impact of the curve’s direction, both left-turning and
right-turning curve geometries were utilized.

####  IV-A2 T-junction

The second template used in this study represents a T-junction, which serves
as a simulation of a road junction area. The parameters varied under this
template include the intersection angle, lane width and junction gap. The
latter describes the size of the actual junction measured from the virtual
crossing point of all junction arms. The primary objective is to gain deeper
insights into low speed turning maneuvers at the intersection. To achieve
this, simulations are conducted for both left-turning and right-turning
scenarios to comprehensively assess the ADF’s behavior in diverse situations.

####  IV-A3 Complex Road

A combination of a T-junctions and a curved road is used to form a complex
road network. The main goal of this is to test the ADF under challenging road
conditions.

###  IV-B Automated Simulation Framework

Once the scenario database is established, the subsequent stage entails
selecting individual scenarios from the database to create a concrete test
case for the automation framework. For each logical scenario, to create
corresponding concrete simulation scenarios, the template OpenSCENARIO file
with default settings is imported, and then scenario-specific information
(e.g., scenario name) is added to the template to generate the desired
concrete OpenSCENARIO file. The road name in the OpenSCENARIO file indicates
the OpenDRIVE map required for simulation. The start road, lane ID, and
position define the initial location where the ego vehicle is spawned, while
the target road, lane ID, and position determine the destination on the
OpenDRIVE map. To streamline this process automatically, a Python script is
used to initialize simulation settings. This script is seamlessly integrated
into the simulation pipeline, ensuring efficient and consistent configuration
of test cases for evaluation.

The utilized system-under-test for conducting simulations is a prototypical
ADF based on [18]. As the ”Simulation” block of Fig. 1 shows it is connected
via ROS and integrated in a closed-loop manner with the previously described
CARLA ROS bridge. An automated simulation framework has been designed to
efficiently run scenarios from the scenario database. The process involves
selecting a scenario, generating its corresponding OpenSCENARIO file, and
executing the simulation facilitated by CARLA ScenarioRunner. A successful
simulation occurs when the ego vehicle reaches its destination and comes to a
stop. To avoid endless attempts for failed scenarios, a predetermined limit is
set on the number of simulation attempts. Simulation data is recorded and
stored in the results database. After completing all simulations, the CARLA
metrics module is employed to extract essential data for KPI calculation from
the recorded files to an independent format, such as CSV. The automation
framework is developed in a highly generic manner to enable easy utilization
for users with their ADF. Minor adjustments to system-specific parameters
allow seamless integration, making the framework accessible to a wide range of
users.

###  IV-C Simulation Evaluation

After completing all simulations, the next step involves analyzing the data to
evaluate the ADF’s performance. Various KPIs have been computed from the
extracted vehicle data, including Longitudinal Acceleration, Longitudinal
Deceleration, Lateral Acceleration, Longitudinal Jerk, Lateral Jerk, and
Distance Target. These metrics are normalized using a reference value within
the range of [$0$, $1$]. A KPI value of $0$ signifies very poor performance,
while a value of $1$ indicates good performance. This normalization process
ensures a standardized evaluation across all criteria, providing a clear and
consistent understanding of the ADF’s performance level, ranging from poor to
good.

The reference values used for evaluating the vehicle’s longitudinal motion are
based on the performance requirements outlined in the ACC system of ISO15622
[19]. According to these requirements, the reference value for longitudinal
acceleration is established at $2\,\text{m}/\text{s}^{2}$, and the reference
value for longitudinal deceleration is set at $3.5\,\text{m}/\text{s}^{2}$.
Additionally, the reference value for vehicle lateral acceleration is defined
as $3\,\text{m}/\text{s}^{2}$, while for longitudinal and lateral jerk, it is
set at $5\,\text{m}/\text{s}^{3}$. These reference values serve as benchmarks
to assess the ADF’s performance in longitudinal and lateral motions, ensuring
adherence to standardized criteria. These KPIs are also referred to as dynamic
KPIs in further sections.

Additionally, comfort-based analysis was conducted on the ADF to address surge
and oscillations in motion planning and control systems. Surge refers to
engine power variations under steady throttle, causing speed fluctuations
during straight-line driving, while oscillations in specific frequency ranges
(e.g., $1$ to $2\,\text{Hz}$) can affect passenger comfort. Vroom’s comfort
analysis approach [20] uses the Root Mean Square (RMS) value of acceleration
based on ISO2631-1:1997 [21] guidelines to evaluate comfort. It presents
acceleration RMS values based on an 8-hour exposure as a comfort reference as
shown in Tab. II. Vertical vibrations were not considered in this study due to
the absence of road height profiles.

Table II: Comfort perception to vibrating environments taken from [21] used to assess the comfort level. Acceleration (x-direction) [$m/s^{2}$] | Perceived comfort
---|---
$\leq 0.314$ | not uncomfortable
$0.315\text{ to }0.63$ | a little uncomfortable
$0.5\text{ to }1.0$ | fairly uncomfortable
$0.8\text{ to }1.6$ | uncomfortable
$1.25\text{ to }2.5$ | very uncomfortable
$\geq 2.0$ | extremely uncomfortable

The acceleration RMS value represents the average longitudinal acceleration
over a specific time period. It is calculated using the variable $\tilde{a}$,
which is the weighted vehicle acceleration in $\text{m}/\text{s}^{2}$ as shown
in (1). The acceleration is filtered through a band-pass filter with a
bandwidth of $1$ to $32$ Hz. The starting time is denoted as ${t_{0}}$ and
${t_{f}}$ represents the final time.

$RMS=\sqrt{\frac{\int_{t_{f}}^{t_{0}}\tilde{a}^{2}\,dt}{t_{f}-t_{0}}}$ (1)

##  V Results

Fig. 4 shows the overall performance of the ADF on $1460$ variations for three
templates used in this study. The average KPIs of curved road, T-junction, and
complex road are $0.737$, $0.816$, and $0.605$ respectively. In the case of
the curved road scenario, the vehicle encountered difficulties at certain
curve radii for various lane widths present in the database. However, the ADF
demonstrated the ability to consistently maintain the lane center throughout
the journey. In the T-junction scenario, the vehicle efficiently reached a
steady state and decelerated in time before approaching the turn at the
intersection. This resulted in stable performance, with the vehicle
maintaining the lane center and demonstrating good overall dynamic
performance. The incorporation of curved road and T-junction in the complex
road template led to numerous crashes, especially in sharp curves and narrow
lane widths. These crashes directly contributed to the poor overall
performance observed in the KPI plot. In this section, the analysis of the
three templates used in this study for the evaluation of the ADF will be
discussed in more detail.

StandstillTimeOscillationsLongitudinalAccelerationLongitudinalDecelerationLateralAccelerationLongitudinalJerkLateralJerkDistanceDrivableSpaceDistanceTarget0.250.50.75CurveT-
junctionComplex Figure 4: Illustration of the overall performance evaluation
for all executed simulations clustered by the three main templates. The spider
plot indicates that the ADF generally have good jerk values and almost always
stays within the lane boundaries. Reaching the target seems to be an issue
which might be controller related since the position is overshot. In the
complex road network, acceleration and steady lane keeping values tend to be
bad which may be due to the transitions between different road elements.

###  V-A Curved Road

The curved road template includes both left and right turning scenarios. For
each set of variation in both scenarios, the defined KPIs are calculated.The
average KPI value for left-turning scenarios was found to be $0.665$, while
for right-turning scenarios, it was $0.832$. Remarkably, the performance
witnessed a substantial $20\%$ decline during the transition from right-
turning to left-turning simulations. Furthermore, through failed simulations,
a critical radius has been identified for left turning scenarios. This
critical radius represents the curve’s radius below which the ADF failed to
accomplish the specified objective. Overall, $6$ simulations failed in left
turning scenario and critical radii of approximately $86\,\text{m}$,
$118\,\text{m}$, and $177\,\text{m}$ for lane width of $4.0\,\text{m}$,
$3.75\,\text{m}$, and $3.5\,\text{m}$ respectively. On the other hand, all
simulations were successful in right turning scenario. The dynamic KPI (cf.
Sec. IV-C) values for three fixed lane widths and varied curve radii in left
curves is shown in Fig. 5. The improved performance with respect to the curve
radius can be clearly seen. The data indicate a positive trend in dynamic KPI
as the curve radius increases in each lane width scenario. In addition, the
performance tends to be better with wider lane width.

The oscillation KPI shown in Fig. 4 can be interpreted such that in a
continuous drive through curves, the ADF maintains a stable and steady driving
trajectory.

$\displaystyle{100}$$\displaystyle{150}$$\displaystyle{200}$$\displaystyle{250}$Curve
radius
[m]$\displaystyle{0.6}$$\displaystyle{0.7}$$\displaystyle{0.8}$$\displaystyle{0.9}$$\displaystyle{1.0}$$\displaystyle{1.1}$$\displaystyle{1.2}$Average
dynamic KPILane width 4.0mLane width 3.75mLane width 3.5m Figure 5: Filtered
average of the dynamic KPIs in left turns with three fixed lane width over
different curve radii. For each lane width, a critical radius was identified
below which the ADF failed to succeed (dashed vertical lines). Expectable, the
performance in wider lanes seems to be better.

###  V-B T-junction

Overall, the ADF performed well on variations of T-junction. Due to the
presence of low speed turning maneuver, the KPI values are observed in good
region. To understand the comfort throughout the journey, mean RMS value of
longitudinal acceleration has been calculated and is shown in Fig. 6 which are
based on the study results from Tab. II. The overlaps are estimated towards
the worse value, respectively. The plot indicates that with increase in
junction gap, the comfort perception also increases. Due to the long turning
section based on the direction of the turn, good performance was observed at
60° in left turning scenario and 120° in right turning scenario.

When looking at the longitudinal deceleration KPI shown in Fig. 4 it can be
seen that the ADF tends to smoothly decelerate when approaching and passing an
intersection. However, the lateral acceleration has room for improvement.

$\displaystyle{5}$$\displaystyle{10}$$\displaystyle{15}$$\displaystyle{20}$$\displaystyle{25}$$\displaystyle{30}$Junction
Gap
[m]$\displaystyle{0.0}$$\displaystyle{0.1}$$\displaystyle{0.2}$$\displaystyle{0.3}$$\displaystyle{0.4}$$\displaystyle{0.5}$$\displaystyle{0.6}$RMS
ValueNot uncomfortableA littleuncomfortable60° Right90° Right120° Right60°
Left90° Left120° Left Figure 6: Average comfort levels with respect to
longitudinal acceleration of three fixed intersection angles. The boundaries
are based on Tab. II. A minor trend to more comfort with increasing junction
gap can be noted. Also, left turns seem to result in more comfort which might
be due to the larger radius on the junction.

###  V-C Complex Road

The complex road, being a combination of curved road and T-junction,
significantly affects the performance of the ADF. The presence of sharp curves
at the T-junction and curve section leads to pronounced lateral and
longitudinal jerk, impacted the overall performance of the ADF. Additionally,
the vehicle’s behavior of slowing down at curves and rapidly accelerating
contributed to lower overall performance in longitudinal and lateral
acceleration KPIs. The overall KPI plot of all three templates used in this
study is shown in Fig. 4, indicating that the combination of multiple
concatenated road geometry affects the ADF’s performance. Also, these
transitions between junctions and connection roads might be the reason for the
multiple occurring oscillations throughout the scenarior runs.

##  VI Conclusion and Outlook

In this research, scenario-based testing was implemented to study the
influence of road geometry parameters on the performance of an automated
driving function. The evaluation revealed that the ADF performed better in
right-turning scenarios compared to left-turning scenarios in the curved road
template. The T-junction template exhibited excellent performance in both left
and right-turning scenarios. However, challenges arose in the complex road
template, resulting in crashes, especially in sharp curves and narrow lanes.
These findings confirm that road geometry significantly influences the ADF’s
performance, aligning with the primary objective of this study.

Looking ahead, further research can focus on enhancing the road network
generation tools to better handle complex road scenarios with lane widening
and additional lanes. Incorporating road height parameters will also enhance
the realism of simulations for urban traffic scenarios with ramps and varying
elevations. Increasing the number of generated road networks and exploring
multiple variables through machine learning techniques can lead to more
diverse and comprehensive scenario variations.

## Acknowledgment

The work of this paper has been done in the context of the SUNRISE project
which is co-funded by the European Commission’s Horizon Europe Research and
Innovation Programme under grant agreement number 101069573. We would like to
express our sincere gratitude to Professor Dr.-Ing. Thomas Esch for his
invaluable support and guidance in publishing this paper.

Views and opinions expressed, are those of the author(s) only and do not
necessarily reflect those of the European Union or the European Climate,
Infrastructure and Environment Executive Agency (CINEA). Neither the European
Union nor the granting authority can be held responsible for them.

## References

  * [1] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. Pratt, M. Sokolsky, G. Stanek, D. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in _2011 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, June 2011.
  * [2] H. Winner and W. Wachenfeld, “Absicherung automatischen fahrens,” in _6\. FAS-Tagung München 29. Nov._ , 2013.
  * [3] ISO, “Road vehicles - safety of the intended functionality (sotif),” International Organization for Standardization (ISO), Tech. Rep., 2022. [Online]. Available: https://www.iso.org/standard/77490.html
  * [4] A. Zlocki, L. Eckstein, and F. Fahrenkrog, “Evaluation and sign-off methodology for automated vehicle systems based on relevant driving situations,” _Transportation Research Record: Journal of the Transportation Research Board_ , vol. 2489, no. 1, pp. 123–129, jan 2015.
  * [5] VMAD, _New Assessment/Test Method for Automated Driving (NATM) Guidelines for Validating Automated Driving System (ADS)_ , 2022. [Online]. Available: https://unece.org/sites/default/files /2022-05/WP.29-187-08e.pdf
  * [6] C. Amersbach and H. Winner, “Defining required and feasible test coverage for scenario-based validation of highly automated vehicles,” in _2019 IEEE Intelligent Transportation Systems Conference (ITSC)_. IEEE, oct 2019.
  * [7] S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, “Survey on scenario-based safety assessment of automated vehicles,” _IEEE Access_ , vol. 8, pp. 87 456–87 477, 2020.
  * [8] _Richtlinien für die Anlage von Autobahnen: RAA_ , Ausgabe 2008 ed., ser. FGSV R1 - Regelwerke. Köln: FGSV-Verl., 2008, vol. 202.
  * [9] M. Scholtes, L. Westhofen, L. R. Turner, K. Lotto, M. Schuldes, H. Weber, N. Wagener, C. Neurohr, M. H. Bollmann, F. Kortke, J. Hiller, M. Hoss, J. Bock, and L. Eckstein, “6-layer model for a structured description and categorization of urban traffic and environment,” _IEEE Access_ , vol. 9, pp. 59 131–59 147, 2021.
  * [10] S. K. Wagner, “Efficient scenario-based assessment of automated driving systemsthrough virtual testing,” 2021. [Online]. Available: https://mediatum.ub.tum.de/doc/1554029/1554029.pdf
  * [11] S. Tak, S. Kim, H. Yu, and D. Lee, “Analysis of relationship between road geometry and automated driving safety for automated vehicle-based mobility service,” _Sustainability_ , vol. 14, no. 4, p. 2336, feb 2022.
  * [12] D. Becker, G. Küppers, and L. Eckstein, “Simulative performance analysis of an ad function with road network variation,” in _15th ITS European Congress, Lisbon, Portugal, 22-24 May-2023_. ITS Europe, 2023.
  * [13] D. Becker, C. Geller, and L. Eckstein, “Road network variation based on hd map analysis for the simulative safety assurance of automated vehicles,” in _2022 International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)_ , 2022, pp. 1–6.
  * [14] F. Poggenhans, J.-H. Pauls, J. Janosovits, S. Orf, M. Naumann, F. Kuhnt, and M. Mayr, “Lanelet2: A high-definition map framework for the future of automated driving,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, nov 2018.
  * [15] D. Becker, F. Russ, C. Geller, and L. Eckstein, “Generation of complex road networks using a simplified logical description for the validation of automated vehicles,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, sep 2020.
  * [16] S. Maierhofer, M. Klischat, and M. Althoff, “CommonRoad scenario designer: An open-source toolbox for map conversion and scenario creation for autonomous vehicles,” in _2021 IEEE International Intelligent Transportation Systems Conference (ITSC)_. IEEE, sep 2021.
  * [17] A. Dosovitskiy, “Carla: An open urban driving simulator,” 2017. [Online]. Available: https://carla.org
  * [18] G. Kueppers, L. Kloeker, J.-P. Busch, A. Kloeker, P. Peltzer, F. Thomsen, S. Klaudt, and L. Eckstein, “Future mobility applications in the komodnext and accord digital test fields,” _31\. Aachen Colloquium Sustainable Mobility_ , October 2022.
  * [19] _ISO 15622:2018 Intelligent transport systems - Adaptive cruise control systems - Performance requirements and test procedures_ , ISO, 2018. [Online]. Available: https://www.iso.org/standard/71515.html
  * [20] N. Vroom, “Jerk based motion planning,” Master’s thesis, TU Delft, 2021. [Online]. Available: http://resolver.tudelft.nl/uuid:c167a041-b1be-493f-b389-d8dad0141a3b
  * [21] “Iso 2631-1:1997 mechanical vibration and shock - evaluation of human exposure to whole-body vibration part 1: General requirements,” International Organization for Standardization (ISO), Tech. Rep., 2021. [Online]. Available: https://www.iso.org/standard/7612.html