11institutetext: FZI Research Center for Information Technology, 76131
Karlsruhe, Germany 11email<EMAIL_ADDRESS>
22institutetext: Karlsruhe Institute of Technology, 76131 Karlsruhe, Germany
33institutetext: Technical University of Berlin, 10623 Berlin, Germany

# AnoVox: A Benchmark for Multimodal
Anomaly Detection in Autonomous Driving

Daniel Bogdoll* 1122 0000-0003-0432-4937 Iramm Hamdard 1122 Lukas Namgyu
Rößler 22 Felix Geisler 22 Muhammed Bayram 22 Felix Wang 22 Jan Imhof 1122
Miguel de Campos 33 Anushervon Tabarov 1122 Yitian Yang 11 Hanno Gottschalk 33
0000-0003-2167-2028 J. Marius Zöllner 1122 0000-0001-6190-7202

###### Abstract

The scale-up of autonomous vehicles depends heavily on their ability to deal
with anomalies, such as rare objects on the road. In order to handle such
situations, it is necessary to detect anomalies in the first place. Anomaly
detection for autonomous driving has made great progress in the past years but
suffers from poorly designed benchmarks with a strong focus on camera data. In
this work, we propose AnoVox, the largest benchmark for ANOmaly detection in
autonomous driving to date. AnoVox incorporates large-scale multimodal sensor
data and spatial VOXel ground truth, allowing for the comparison of methods
independent of their used sensor. We propose a formal definition of normality
and provide a compliant training dataset. AnoVox is the first benchmark to
contain both content and temporal anomalies.

###### Keywords:

Anomaly Detection Autonomous Driving Benchmark

\textsuperscript{\textasteriskcentered}\textsuperscript{\textasteriskcentered}footnotetext:
These authors contributed equally

##  1 Introduction

In the past years, we have seen autonomous vehicles scale from small and well-
defined Operational Design Domains (ODD) to city-wide areas, potentially
reaching geographic areas covering multiple neighboring cities in the near
future [73, 75, 74]. However, with a growing fleet in an expanding ODD,
scenarios from the long tail of rare events [71] occur more frequently as a
consequence of increased exposure. Expert perspectives, focusing on content
and temporal anomalies, are commonly used to judge "long tail" data points
[18, 17, 40, 10, 11, 61, 66], as replaying training data during inference is
not possible. Such expert views are often used for the creation of anomaly
detection benchmarks but are of no help for anomaly detection. Few
publications take a data-oriented perspective on the definition of anomalies
[39, 25, 82], also known as out-of-distribution (OOD), corner cases, or
outliers.

Recent research has focused on anomaly detection methods [42] and its
ramifications, such as OOD segmentation [58], object detection [31], instance
segmentation [55], and video tracking [54]. These methods typically do not
learn a representation of normality from training data alone, but require a
semantic segmentation stage and fine-tuning on auxiliary data containing
(artificial) anomalies [9, 23]. While there exists a considerable number of
datasets in autonomous driving [53, 49, 14], only around 5 - 10% of datasets
are designed for benchmarking anomaly detection methods in autonomous driving
[15, 13]. These existing datasets, however, have significant limitations.
First, the majority are camera-only, despite autonomous driving relying on
multi-modal sensor setups. Second, temporal information is often overlooked,
limiting anomaly detection to single frames. Additionally, content anomalies
are rarely found in challenging traffic situations, which allows for the
assumption that anything on the road might be an anomaly [54]. Most anomalies
are human-defined, such as dogs on the street [9], making detection
challenging as these classes are typically included in training datasets but
not annotated [27, 31]. As a result, anomaly detection methods can miss
anomalies they don’t perceive as atypical due to their training.

Figure 1: Overview of AnoVox. AnoVox includes both content and temporal
anomalies. Here, we see an example of a content anomaly. Ground truth is
available for both camera and lidar, enabling the application of anomaly
detection methods for both modalities. We additionally provide ground truth in
a voxelized form. By mapping the anomaly scores to this voxelized space, we
can compare methods from any sensor.

With AnoVox, we offer a challenging and adaptable benchmark for ANOmaly
detection, leveraging the CARLA simulation engine [30] to provide ground truth
data in all modalities and spatial VOXel representation. Given the surge in
foundation models [21, 60], world models [80, 45], and large-scale pre-
training approaches [43, 78], we see potential for the application of self-
and unsupervised training paradigms in the field of anomaly detection as well.
Hence, we prioritize support for label-free training paradigms and provide an
extensive synthetic dataset to enable research in OOD segmentation, addressing
the limitations of current datasets and enabling rapid prototyping for
technologies that fill the blind spots of current anomaly detection methods.
Our contributions include:

  1. 1.

Formal definition of normality and provision of a compliant training dataset
for a fair comparison of anomaly detection methods

  2. 2.

Large and challenging benchmark with a high scene complexity and temporal
scenarios that include many frames without anomalies

  3. 3.

First benchmark to provide content and temporal anomalies, adaptable
multimodal sensor data, ego-vehicle state information, and ground truth in all
modalities, as well as a spatial voxel representation

All code to reproduce our results, create new AnoVox instantiations, and
benchmark anomaly detection methods can be found on GitHub. The dataset can be
found on Zenodo.

##  2 Related Work

Table 1: Vision-based anomaly detection benchmarks. In the Normality column, †
denotes a domain shift between normal data and the proposed dataset, and ⊛
denotes that the data with anomalies is based on a subset of the normal data.
Adapted from [15].

Dataset | Data | Size |  | Ano.
---
Source
| Ano.
---
Type
| #Ano.
---
Classes
| Ground
---
truth
| Temp.
---
Data
| Ego
---
Act.
| Reg.
---
Tasks
Normality
Fishyscapes [8, 9] |  |  |  |  |  |  |  |  |  |
FS Lost and Found | Cam. | 375 | Recording | Content | 1 | Sem. mask (2D) | — | — | — | —
FS Static | Cam. | 1,030 | Data Augmentation | Content | 1 | Sem. mask (2D) | — | — | — | Cityscapes⊛
Crash to Not Crash [46] |  |  |  |  |  |  |  |  |  |
YouTubeCrash | Cam. | 2,400 | Web Sourcing | Temporal | 1 | Bbox (2D) | ✓ | — | ✓ | YouTubeCrash
GTACrash | Cam. | 154,400 | Simulation | Temporal | 1 | Bbox (2D) | ✓ | — | ✓ | GTA V
CAOS [41] |  |  |  |  |  |  |  |  |  |
StreetHazards | Cam. | 1,500 | Simulation | Content | 1 | Sem. mask (2D) | ✓ | — | ✓ | CARLA†
BDD-Anomaly | Cam. | 810 | Class Exclusion | Content | 3 | Sem. mask (2D) | — | — | ✓ | BDD100K⊛
SegmentMeIfYouCan [23] |  |  |  |  |  |  |  |  |  |
RoadAnomaly21 | Cam. | 110 | Web Sourcing | Content | 1 | Sem. mask (2D) | — | — | — | Cityscapes†
RoadObstacle21 | Cam. | 412 | Recording | Content | 1 | Sem. mask (2D) | ✓ | — | — | Cityscapes†
Rare Road Objects [19] |  |  |  |  |  |  |  |  |  |
Synthetic Fire Hydrants | Cam. | 30,000 | Simulation | Content | 1 | Bbox (2D) | — | — | — | CARLA†
Synthetic Crosswalks | Cam. | 20,000 | Simulation | Content | 1 | Bbox (2D) | — | — | — | CARLA†
CODA [51] |  |  |  |  |  |  |  |  |  |
CODA-KITTI | Cam., Lidar | 309 | Void Classes | Content | 6 | Bbox (2D) | — | — | — | KITTI⊛
CODA-nuScenes | Cam., Lidar | 134 | Void Classes | Content | 17 | Bbox (2D) | — | — | — | nuScenes⊛
CODA-ONCE | Cam., Lidar | 1,057 | OOD Detection | Content | 32 | Bbox (2D) | — | — | — | ONCE⊛
CODA2022-ONCE | Cam., Lidar | 1,057 | OOD Detection | Content | 29 | Bbox (2D) | — | — | ✓ | ONCE⊛
CODA2022-SODA10M | Cam. | 8,711 | OOD Detection | Content | 29 | Bbox (2D) | — | — | ✓ | SODA10M⊛
W-OOD Tracking [54] |  |  |  |  |  |  |  |  |  |
Street Obstacle Sequences | Cam., Depth |  | 1,129
---
Recording | Content | 13 | Inst. mask (2D) | ✓ | — | — | Cityscapes†
CARLA-WildLife | Cam., Depth | 1,210 | Simulation | Content | 18 | Inst. mask (2D) | ✓ | — | ✓ | CARLA
Misc |  |  |  |  |  |  |  |  |  |
Lost and Found [62] | Stereo Cam. | 2,104 | Recording | Content | 42 | Sem. mask (2D) | ✓ | — | — | —
WD-Pascal [6] | Cam. | 70 | Data Augmentation | Content | 1 | Sem. mask (2D) | — | — | — | WildDash⊛
Vistas-NP [36] | Cam. | 11,167 | Class Exclusion | Content | 4 | Sem. mask (2D) | — | — | ✓ | Mapillary Vistas⊛
MUAD [32] | Cam., Depth | 4,641 | Sim., Class Exclusion | Content | 9 | Sem. mask (2D) | — | — | ✓ | MUAD
DeepAccident [72] | Cam., Lidar | 57,000 | Simulation | Temporal | 9 | Bbox (3D) | ✓ | — | ✓ | CARLA
AnoVox (ours) |  | Cam., Lidar
---
Depth
245,600 | Simulation |  | Content
---
Temporal
178 |  | Inst. mask (2D,3D)
---
Voxel (3D)
✓ | ✓ | ✓ | CARLA

In autonomous driving, anomaly detection has been classically evaluated on a
small set of benchmarks. For our analysis of related works, we have included
works with published open-access perception datasets from an ego perspective
that provide pixel- or point-wise ground truth. Thus, we neglected frameworks
that do not provide explicit data [77, 68, 38, 76, 65], such with missing or
incomplete data [34, 52, 48], and any that provide only frame-wise annotations
[67, 79, 70, 2]. For our overview of related works as shown in Table 1, we
updated and extended the survey of Bogdoll et al. [15], comparing now 22
datasets from the literature.

Figure 2: Overview: With Anovox, highly configurable scenarios can be created
that either represent normality or include content or temporal anomalies.
Generated datasets include rich labels and ground truth in 2D, 3D, and a
spatial voxel space.

It can be observed that most benchmarks are small and designed for camera-
based content anomaly detection, providing ground truth in the form of
semantic masks. Among the datasets including content anomalies, just the CODA
[51] family includes lidar data but only provides ground truth in the form of
2D bounding boxes in the camera space. The DeepAccident [72] benchmark is the
only one to provide temporal 3D labels for lidar point clouds.

There are different categories of how anomalies were introduced [15] in the
datasets. Recording and Simulation are similar in the way that selected
anomalies were directly introduced into the data. This way, the anomalies are
truly part of the environment [62, 19, 54]. The definition of what counts as
an anomaly can vary, though. Data Augmentation typically follows a copy-and-
paste pattern, where images of anomalies are pasted onto an already existing
scene from another dataset. This way, a distribution shift between the anomaly
and the underlying data is introduced [9, 6]. Web Sourcing describes the
process of manually curating images from the web that are deemed anomalous
[23, 46]. Class Exclusion is based on existing datasets and removes selected
classes from the training data, thus treating them as anomalous, while the
classes themselves remain rather normal from a human point of view [35, 41].
Void Classes utilizes void or misc classes from existing datasets and labels
them as anomalous. This can be done with additional labeling guidelines to
only relabel selected ones [51]. Finally, OOD Detection uses an anomaly
detection method to derive anomaly proposals from a dataset which can then be
labeled, typically after a human quality inspection [51].

While many datasets include temporal data in the form of sequences and provide
labels for regular tasks, such as object detection or semantic segmentation,
none includes state information about the ego vehicle. However, during
deployment, such information is generally available and can be leveraged.

Most benchmarks do not provide a definition of normality [9, 23, 41]. This
makes it especially hard for un- and self-supervised methods to precisely
detect anomalies if the semantic training distribution is not fully known.
Especially desirable is a well-defined normality that allows for the
generation of compliant training data. This is only possible in simulation,
where full control over both the training and evaluation data is available.
Sometimes, even unlabeled anomalies occur in evaluation data [72].

The Synthetic Dataset for Anomaly and Corner Case Detection (SDAC) by Gong et
al. [34] was developed concurrently to AnoVox and shows the highest
similarities to our work. They provide multimodal camera and lidar data from
the CARLA environment. Labels are provided in the form of 2D and 3D bounding
boxes as well as panoptic segmentation masks.

They provide a small and fixed dataset representing normality. However, their
definition is very narrow and only consists of "clear, sunny weather"
scenarios with four normal classes: car, motorcycle, bicycle, and person [34].

They provide content and temporal anomalies. Their content anomalies in the
"TruckBus" set are no true anomalies, but the withheld classes truck and bus.
The "Props" subset includes small objects, like traffic cones, or boxes. The
"Accident" subset combines unknown weather conditions with known objects in
atypical positions, such as overturned vehicles or pedestrians who are lying
down. In all cases, the anomalies are spawned within the drivable area of the
frames.

In addition, they also provide scene anomalies that include known objects with
weather shifts. However, as we do not focus on frame-wise labels, they are
irrelevant here.

Their scenario anomalies focus on "pre-crash and crash scenarios" [34]. They
provide seven manually generated scenario types, also including sudden
breaking scenarios.

"Download link for SDAC-Scenario: [Comming Soon]" –> ignore because
unpublished!

AnoVox is a challenging benchmark that addresses the field of anomaly
detection from an embodied AI perspective rather than from a pure computer
vision perspective. Therefore, we provide data in a temporal context with
state information about the ego vehicle for typical multimodal sensor setups.
For a clear definition of what anomalies are, we provide a formalized
definition of normality and provide compliant training data. This especially
allows for the usage of un- and self-supervised anomaly detection methods,
which often rely on large amounts of unlabeled data.

##  3 AnoVox Dataset

Table 2: A formal definition of normality and how the training data generated
and provided by AnoVox aligns with the definition.

Category | Description | AnoVox
---|---|---
Ego |  |
Ego vehicle | Recording vehicle | Lincoln MKZ 2020
Sensor config. | Sensor types, placements | Configurations Mono, Stereo, Multi, Surround
Ego behavior | Driving characteristics | Behavior Agent (actions)
Domain |  |
Area | Geographical area | Towns 01,02,03,04,05,06,07,10HD
Environment | Weather, time of day |  | ClearNoon, CloudyNoon, WetNoon,
---
WetCloudyNoon, HardRainNoon,
SoftRainNoon, ClearSunset, CloudySunset,
WetSunset, WetCloudySunset, MidRainSunset,
HardRainSunset, SoftRainSunset
Physical entities |  |
Traffic participants | Vehicles, VRUs | Vehicles, Walkers $\in$ Blueprint library
Vehicle behavior | Driving characteristics | Traffic Manager Autopilot
Pedestrian behavior | Movement characteristics | AI Walker

AnoVox provides both data that represents normality and data including content
and temporal anomalies. While we do provide a large-scale dataset, AnoVox is
primarily a scalable framework that can be used with arbitrary vehicle setups
and a wide selection of parameters to create further data. This does not only
allow for the detection of anomalies in known environments but also for the
detection of anomalies under domain shifts. In Sec. 3.1, we provide a
formalized definition of normality and demonstrate how the training data
provided by AnoVox follows this definition of normality. In Sec. 3.2, we show
what types of scenarios can be generated with AnoVox. Finally, Sec. 3.3
provides an overview of our dataset.

###  3.1 Definition of Normality

In the past, the definition of normality was often not extensively discussed
when anomaly detection benchmarks were presented. A typical solution is to
define normality as all semantic classes from the Cityscapes dataset [9, 23].
However, this is not necessarily related to the training data. And if it is,
it requires labeled training data as normality to be aware of the classes.

With AnoVox, we provide full controllability about both normality and
anomalies in synthetic environments. This ensures that anomalies included in
the benchmark are true anomalies and are not included in an unlabeled training
dataset by chance. For a fair benchmark of anomaly detection methods, it is
important that they share the same definition of normality. We argue that this
normality needs to be primarily defined by the training data rather than
expert-defined concepts. This, however, requires the option to generate large
amounts of training data following a definition of normality, which is
infeasible in the real world, as anomalies would certainly occur in fleet-
sized, unlabeled datasets. However, in the field of autonomous driving as a
subfield of embodied AI, there is more to the training data than just frames:
There is a recording entity that performs actions, and there is temporal
context. Therefore, we provide a formal definition of normality based on three
categories:

Ego: Domain shifts in data cannot only be induced by novel environmental
conditions but also by different capturing methods. In autonomous driving,
this especially refers to sensor types and configurations. In addition,
temporal changes in the environment are heavily influenced by our own actions.
Thus, also the behavior of the ego agent counts towards normality.

Domain: With the domain, we describe the static environment around the
vehicle. This includes the geographical areas the vehicle has traversed, but
also seen weather types and time of day specifications.

Physical Entities: These are the dynamic actors in the scene, most typically
other vehicles, cyclists, and pedestrians. However, also categories such as
animals or potentially moving objects can be included here.

Based on these concepts, we provide a definition of normality for the CARLA
simulation engine in Table 2 that allows for the generation of compliant
datasets.

4.661766in1.258477inuse as bounding box,
clipcurrentfill0.000000ptcurrentstroke0pt0.000000in0.000000in4.661766in0.000000in4.661766in1.258477in0.000000in1.258477in0.000000in0.000000infillcurrentfill0.000000ptcurrentstroke0.0000000pt0.000000in0.000000in1.059404in0.000000in1.059404in1.059404in0.000000in1.059404in0.000000in0.000000infill0.000000in0.000000in1.059404in1.059404inclipcurrentfill0.5000000.000000ptcurrentstroke0.5000000pt0.504607in0.461079in0.554796in0.461079in0.561418in0.461079in0.561418in0.467700in0.561418in0.591703in0.561418in0.598325in0.554796in0.598325in0.504607in0.598325in0.497986in0.598325in0.497986in0.591703in0.497986in0.467700in0.497986in0.461079in0.504607in0.461079in0.504607in0.461079infill0.000000in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt0.529702in0.264851in0.599941in0.264851in0.667313in0.292757in0.716980in0.342424in0.766646in0.392091in0.794553in0.459463in0.794553in0.529702in0.794553in0.599941in0.766646in0.667313in0.716980in0.716980in0.667313in0.766646in0.599941in0.794553in0.529702in0.794553in0.459463in0.794553in0.392091in0.766646in0.342424in0.716980in0.292757in0.667313in0.264851in0.599941in0.264851in0.529702in0.264851in0.459463in0.292757in0.392091in0.342424in0.342424in0.392091in0.292757in0.459463in0.264851in0.529702in0.264851in0.529702in0.264851instroke0.803000ptcurrentstroke0pt0.000000in0.000000in0.000000in1.059404instroke0.803000ptcurrentstroke0pt1.059404in0.000000in1.059404in1.059404instroke0.803000ptcurrentstroke0pt0.000000in0.000000in1.059404in0.000000instroke0.803000ptcurrentstroke0pt0.000000in1.059404in1.059404in1.059404instroke0.000000in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt0.810619in0.810619in0.736143in0.885094in0.635026in0.926978in0.529702in0.926978in0.424378in0.926978in0.323260in0.885094in0.248785in0.810619in0.529702in0.529702in0.810619in0.810619instroketextcolortextcolor[x=0.529702in,y=1.142737in,,base]Mono0.000000in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill0.529702in0.529702incurrentmarkercurrentfill0.000000ptcurrentstroke0.0000000pt1.200787in0.000000in2.260191in0.000000in2.260191in1.059404in1.200787in1.059404in1.200787in0.000000infill1.200787in0.000000in1.059404in1.059404inclipcurrentfill0.5000000.000000ptcurrentstroke0.5000000pt1.705395in0.461079in1.755584in0.461079in1.762205in0.461079in1.762205in0.467700in1.762205in0.591703in1.762205in0.598325in1.755584in0.598325in1.705395in0.598325in1.698773in0.598325in1.698773in0.591703in1.698773in0.467700in1.698773in0.461079in1.705395in0.461079in1.705395in0.461079infill1.200787in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt1.730489in0.264851in1.800728in0.264851in1.868100in0.292757in1.917767in0.342424in1.967434in0.392091in1.995340in0.459463in1.995340in0.529702in1.995340in0.599941in1.967434in0.667313in1.917767in0.716980in1.868100in0.766646in1.800728in0.794553in1.730489in0.794553in1.660250in0.794553in1.592878in0.766646in1.543211in0.716980in1.493545in0.667313in1.465638in0.599941in1.465638in0.529702in1.465638in0.459463in1.493545in0.392091in1.543211in0.342424in1.592878in0.292757in1.660250in0.264851in1.730489in0.264851in1.730489in0.264851instroke0.803000ptcurrentstroke0pt1.200787in0.000000in1.200787in1.059404instroke0.803000ptcurrentstroke0pt2.260191in0.000000in2.260191in1.059404instroke0.803000ptcurrentstroke0pt1.200787in0.000000in2.260191in0.000000instroke0.803000ptcurrentstroke0pt1.200787in1.059404in2.260191in1.059404instroke1.200787in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.048055in0.728340in1.995394in0.819553in1.908562in0.886181in1.806827in0.913441in1.705092in0.940701in1.596579in0.926415in1.505366in0.873753in1.704004in0.529702in2.048055in0.728340instroke1.200787in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt1.955612in0.873753in1.910432in0.899838in1.860553in0.916770in1.808829in0.923579in1.757106in0.930389in1.704544in0.926944in1.654152in0.913441in1.603759in0.899939in1.556517in0.876641in1.515128in0.844882in1.473739in0.813123in1.439008in0.773520in1.412923in0.728340in1.756974in0.529702in1.955612in0.873753instroketextcolortextcolor[x=1.730489in,y=1.142737in,,base]Stereo1.200787in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill1.730489in0.529702incurrentmarker1.200787in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill1.704004in0.529702incurrentmarker1.200787in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill1.756974in0.529702incurrentmarkercurrentfill0.000000ptcurrentstroke0.0000000pt2.401575in0.000000in3.460978in0.000000in3.460978in1.059404in2.401575in1.059404in2.401575in0.000000infill2.401575in0.000000in1.059404in1.059404inclipcurrentfill0.5000000.000000ptcurrentstroke0.5000000pt2.906182in0.461079in2.956371in0.461079in2.962992in0.461079in2.962992in0.467700in2.962992in0.591703in2.962992in0.598325in2.956371in0.598325in2.906182in0.598325in2.899561in0.598325in2.899561in0.591703in2.899561in0.467700in2.899561in0.461079in2.906182in0.461079in2.906182in0.461079infill2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.931277in0.264851in3.001516in0.264851in3.068888in0.292757in3.118554in0.342424in3.168221in0.392091in3.196127in0.459463in3.196127in0.529702in3.196127in0.599941in3.168221in0.667313in3.118554in0.716980in3.068888in0.766646in3.001516in0.794553in2.931277in0.794553in2.861037in0.794553in2.793665in0.766646in2.743999in0.716980in2.694332in0.667313in2.666426in0.599941in2.666426in0.529702in2.666426in0.459463in2.694332in0.392091in2.743999in0.342424in2.793665in0.292757in2.861037in0.264851in2.931277in0.264851in2.931277in0.264851instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.931277in0.304579in3.001516in0.304579in3.068888in0.332485in3.118554in0.382152in3.168221in0.431818in3.196127in0.499190in3.196127in0.569429in3.196127in0.639669in3.168221in0.707041in3.118554in0.756707in3.068888in0.806374in3.001516in0.834280in2.931277in0.834280in2.861037in0.834280in2.793665in0.806374in2.743999in0.756707in2.694332in0.707041in2.666426in0.639669in2.666426in0.569429in2.666426in0.499190in2.694332in0.431818in2.743999in0.382152in2.793665in0.332485in2.861037in0.304579in2.931277in0.304579in2.931277in0.304579instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.931277in0.225123in3.001516in0.225123in3.068888in0.253030in3.118554in0.302696in3.168221in0.352363in3.196127in0.419735in3.196127in0.489974in3.196127in0.560213in3.168221in0.627585in3.118554in0.677252in3.068888in0.726919in3.001516in0.754825in2.931277in0.754825in2.861037in0.754825in2.793665in0.726919in2.743999in0.677252in2.694332in0.627585in2.666426in0.560213in2.666426in0.489974in2.666426in0.419735in2.694332in0.352363in2.743999in0.302696in2.793665in0.253030in2.861037in0.225123in2.931277in0.225123in2.931277in0.225123instroke0.803000ptcurrentstroke0pt2.401575in0.000000in2.401575in1.059404instroke0.803000ptcurrentstroke0pt3.460978in0.000000in3.460978in1.059404instroke0.803000ptcurrentstroke0pt2.401575in0.000000in3.460978in0.000000instroke0.803000ptcurrentstroke0pt2.401575in1.059404in3.460978in1.059404instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.212193in0.810619in3.137718in0.885094in3.036600in0.926978in2.931277in0.926978in2.825953in0.926978in2.724835in0.885094in2.650360in0.810619in2.931277in0.529702in3.212193in0.810619instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.212193in0.850346in3.137718in0.924821in3.036600in0.966706in2.931277in0.966706in2.825953in0.966706in2.724835in0.924821in2.650360in0.850346in2.931277in0.569429in3.212193in0.850346instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.153751in0.900238in3.108571in0.926323in3.058692in0.943255in3.006968in0.950064in2.955245in0.956874in2.902683in0.953429in2.852290in0.939926in2.801898in0.926424in2.754656in0.903126in2.713267in0.871367in2.671877in0.839608in2.637147in0.800005in2.611062in0.754825in2.955113in0.556187in3.153751in0.900238instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.251491in0.754825in3.198829in0.846038in3.111998in0.912667in3.010263in0.939926in2.908528in0.967186in2.800015in0.952900in2.708802in0.900238in2.907440in0.556187in3.251491in0.754825instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.650360in0.209057in2.724835in0.134582in2.825953in0.092698in2.931277in0.092698in3.036600in0.092698in3.137718in0.134582in3.212193in0.209057in2.931277in0.489974in2.650360in0.209057instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.629683in0.262106in2.690095in0.175830in2.782403in0.117023in2.886127in0.098733in2.989851in0.080444in3.096705in0.104133in3.182981in0.164544in2.955113in0.489974in2.629683in0.262106instroke2.401575in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt2.679572in0.164544in2.722307in0.134621in2.770520in0.113406in2.821454in0.102115in2.872387in0.090823in2.925049in0.089674in2.976426in0.098733in3.027804in0.107793in3.076897in0.126884in3.120896in0.154915in3.164896in0.182946in3.202946in0.219371in3.232870in0.262106in2.907440in0.489974in2.679572in0.164544instroketextcolortextcolor[x=2.931277in,y=1.142737in,,base]Multi2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.931277in0.529702incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.931277in0.569429incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.955113in0.556187incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.907440in0.556187incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.931277in0.489974incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.955113in0.489974incurrentmarker2.401575in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill2.907440in0.489974incurrentmarkercurrentfill0.000000ptcurrentstroke0.0000000pt3.602362in0.000000in4.661766in0.000000in4.661766in1.059404in3.602362in1.059404in3.602362in0.000000infill3.602362in0.000000in1.059404in1.059404inclipcurrentfill0.5000000.000000ptcurrentstroke0.5000000pt4.106969in0.461079in4.157159in0.461079in4.163780in0.461079in4.163780in0.467700in4.163780in0.591703in4.163780in0.598325in4.157159in0.598325in4.106969in0.598325in4.100348in0.598325in4.100348in0.591703in4.100348in0.467700in4.100348in0.461079in4.106969in0.461079in4.106969in0.461079infill3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.132064in0.264851in4.202303in0.264851in4.269675in0.292757in4.319342in0.342424in4.369008in0.392091in4.396915in0.459463in4.396915in0.529702in4.396915in0.599941in4.369008in0.667313in4.319342in0.716980in4.269675in0.766646in4.202303in0.794553in4.132064in0.794553in4.061825in0.794553in3.994453in0.766646in3.944786in0.716980in3.895119in0.667313in3.867213in0.599941in3.867213in0.529702in3.867213in0.459463in3.895119in0.392091in3.944786in0.342424in3.994453in0.292757in4.061825in0.264851in4.132064in0.264851in4.132064in0.264851instroke0.803000ptcurrentstroke0pt3.602362in0.000000in3.602362in1.059404instroke0.803000ptcurrentstroke0pt4.661766in0.000000in4.661766in1.059404instroke0.803000ptcurrentstroke0pt3.602362in0.000000in4.661766in0.000000instroke0.803000ptcurrentstroke0pt3.602362in1.059404in4.661766in1.059404instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.412981in0.810619in4.338505in0.885094in4.237388in0.926978in4.132064in0.926978in4.026740in0.926978in3.925622in0.885094in3.851147in0.810619in4.132064in0.529702in4.412981in0.810619instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.166689in0.925466in4.061766in0.934646in3.957382in0.901734in3.876700in0.834033in3.796017in0.766332in3.745479in0.669250in3.736299in0.564327in4.132064in0.529702in4.166689in0.925466instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.904196in0.855132in3.817919in0.794720in3.759112in0.702412in3.740823in0.598688in3.722534in0.494964in3.746223in0.388110in3.806634in0.301833in4.132064in0.529702in3.904196in0.855132instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.748324in0.632524in3.734822in0.582132in3.731377in0.529570in3.738186in0.477847in3.744996in0.426123in3.761928in0.376244in3.788012in0.331064in3.814097in0.285883in3.848828in0.246280in3.890217in0.214521in3.931606in0.182762in3.978849in0.159465in4.029241in0.145962in4.132064in0.529702in3.748324in0.632524instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.772009in0.361806in3.794057in0.314524in3.825204in0.272044in3.863668in0.236799in3.902132in0.201554in3.947164in0.174227in3.996187in0.156384in4.045211in0.138541in4.097273in0.130528in4.149393in0.132804in4.201513in0.135079in4.252678in0.147599in4.299960in0.169647in4.132064in0.529702in3.772009in0.361806instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.527828in0.564327in4.523282in0.616298in4.508542in0.666868in4.484452in0.713144in4.460363in0.759419in4.427393in0.800499in4.387428in0.834033in4.347464in0.867567in4.301282in0.892903in4.251527in0.908591in4.201772in0.924279in4.149410in0.930013in4.097439in0.925466in4.132064in0.529702in4.527828in0.564327instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.457494in0.301833in4.517905in0.388110in4.541594in0.494964in4.523305in0.598688in4.505015in0.702412in4.446208in0.794720in4.359932in0.855132in4.132064in0.529702in4.457494in0.301833instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt4.234887in0.145962in4.336622in0.173222in4.423453in0.239851in4.476115in0.331064in4.528777in0.422277in4.543063in0.530789in4.515803in0.632524in4.132064in0.529702in4.234887in0.145962instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.964168in0.169647in4.059623in0.125135in4.168968in0.120361in4.267940in0.156384in4.366912in0.192407in4.447607in0.266350in4.492119in0.361806in4.132064in0.529702in3.964168in0.169647instroke3.602362in0.000000in1.059404in1.059404inclip2.007500ptcurrentstroke0.5000000pt3.851147in0.248785in3.925622in0.174310in4.026740in0.132425in4.132064in0.132425in4.237388in0.132425in4.338505in0.174310in4.412981in0.248785in4.132064in0.529702in3.851147in0.248785instroketextcolortextcolor[x=4.132064in,y=1.142737in,,base]Surround3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker3.602362in0.000000in1.059404in1.059404inclipcurrentfill1.003750ptcurrentstroke0ptcurrentmarker-0.026896in-0.026896in0.026896in0.026896in0.000000in-0.026896in0.007133in-0.026896in0.013974in-0.024062in0.019018in-0.019018in0.024062in-0.013974in0.026896in-0.007133in0.026896in0.000000in0.026896in0.007133in0.024062in0.013974in0.019018in0.019018in0.013974in0.024062in0.007133in0.026896in0.000000in0.026896in-0.007133in0.026896in-0.013974in0.024062in-0.019018in0.019018in-0.024062in0.013974in-0.026896in0.007133in-0.026896in0.000000in-0.026896in-0.007133in-0.024062in-0.013974in-0.019018in-0.019018in-0.013974in-0.024062in-0.007133in-0.026896in0.000000in-0.026896in0.000000in-0.026896instroke,fill4.132064in0.529702incurrentmarker

Figure 3: Preconfigured sensor configurations in the AnoVox suite. Blue wedges
visualize RGB cameras, while red circles visualize lidar sensors.

###  3.2 Scenario Generation

AnoVox is designed to create configurable, large-scale datasets that either
define normality or include anomalies. As shown in Fig. 2, first, the vehicle
sensor configuration needs to be set. AnoVox currently supports an arbitrary
number of camera, lidar, and depth sensors, which can be positioned freely on
the ego vehicle of choice. This allows for the replication of existing sensor
setups, the alignment with other datasets, or the testing of new
configurations.

We provide four pre-designed multimodal vehicle sensor configurations. As
shown in Fig. 3, the mono configuration consists of a lidar and a camera which
are centered on top of the vehicle, centered forward. Our stereo setup
consists of two cameras at the front edge and both a camera and lidar on top.
The multi setup adds rear-facing cameras and additional lidar sensors at the
front and rear, positioned at a lower level compared to the roof-mounted
lidars. Finally, the surround setup provides a full 360° camera view next to a
top-mounted lidar. Every RGB camera automatically comes with a depth camera.

Second, the environment and actors need to be set. AnoVox currently supports
eight different regions and 14 weather and time of day presets. Pedestrians,
cyclists, as well as multiple types of vehicles can be spawned in the scene.

Third, the type of anomaly needs to be defined. Currently, AnoVox supports to
create normality without anomalies, the placement of content anomalies, and
the activation of temporal anomalies. By removing domains from the training
dataset, AnoVox also supports the detection of anomalies under domain shifts.

Given these configurations, AnoVox pre-computes the scenario flows and stores
them as scenario descriptions. Thus, meta-data describing all scenarios is
available and allows for effortless dataset analysis. Scenario descriptions
can be shared among peers by sharing the scenario description files and
locally generating the scenarios without the need to move hundreds of
gigabytes of data.

Based on the given configuration, AnoVox executes the driving scenarios in
simulation. We provide a custom-built CARLA simulation engine [30] including
all content anomalies that we have manually collected and processed. Each
scenario has a length of 20 seconds and is recorded at 10 Hz, resulting in 200
frames. When the scenario starts, the ego vehicle is spawned in the world and
follows a given route to its target goal. At some point along the way, a
content or temporal anomaly will appear. To guarantee that the anomaly is
reached in time, a green wave is activated along the route of the ego vehicle.
Since physics computations remain active in the simulation, the ego vehicle
will make contact with the anomalies, which leads to realistic collisions.

|  |
---|---|---
|  |
Figure 4: Content anomalies:. Examples from our six categories: An ape as an
animal anomaly, an old tv as a home anomaly, a statue as a special anomaly, a
tree as a nature anomaly, a pillar as a falling anomaly, and a hot air balloon
as an airplane anomaly.

####  3.2.1 Actor Routing.

Strange objects on the road do the same thing in simulation as they would do
in real life - they cause traffic jams. This makes it often unfeasible to
reach the anomaly for the ego vehicle. Thus, we deploy a filter and rerouting
algorithm for all other vehicles in our scenario. First, we filter all actors
close to our spawn point and on our direct path toward the anomaly. Then, we
continuously monitor all planned paths and reroute vehicles whenever they were
to enter a lane with an anomaly on it, as they would get stuck. This rerouting
technically changes their driving behavior. As shown in Table 2, in the
training data, only vehicles are present, which show an autopilot driving
behavior. Rerouting makes them switch into a behavior agent. We deal with this
by providing novel labels, as further explained in Sec. 3.2.5. This way, false
positives, which might occur due to a violation of the alignment with
normality, can be filtered out for evaluation.

####  3.2.2 Content Anomalies.

We provide 178 different content anomalies in five different size classes
tiny, small, medium, big, and huge. Semantically, they hierarchically belong
to six different super-classes. Every anomaly has an individual label next to
its super-class and size label.

The class animal includes 33 animals of different sizes. The category home
includes 53 typical household items such as furniture, tables, backpacks, or
cardboard boxes. The category special includes 67 objects of rather atypical
types and such that fall into a misc category, such as Pokemon, some of which
could appear in the real world in the form of costumes or cuddly toys. The
class nature includes 12 outdoor objects, such as rocks or wood. In the
category falling, we provide 9 large objects, such as novel trees, that were
spawned in an unstable position, which made them fall over. Finally, the
airplane class consists of four types of large flying objects.

As especially large anomalies sometimes require manual positioning, we use the
classes home, special, and animal for our automated scenario generation. Here,
we place all anomalies in critical positions along the way distributed around
lane centers. For the classes nature, falling, and airplane, we provide a
manually curated dataset in a geographic area with large free-space areas.

Figure 5: Temporal anomalies: This scenario shows our implemented type of
temporal anomalies. The first two images show the regular vehicle following
mode. The third and fourth images show the active braking maneuver with
overlayed ground truth.

####  3.2.3 Temporal Anomalies.

There are two ways to generate temporal anomalies: Optimization-based or
knowledge-based [72]. While both are of value, for our benchmark, it is
important to comply with the defined normality, which is more challenging with
optimization-based methods. Thus, we provide expert-designed content
anomalies. As prior work has shown that expert-defined temporal anomalies
require high engineering efforts [11, 12, 2], we provide only a single type of
temporal anomalies. We implement sudden braking scenarios of a lead vehicle,
which is a very typical scenario in everyday traffic [46]. While we set a
planned route for the ego vehicle in our content anomaly scenarios, we use the
same route and follow it with a lead vehicle. Then, our ego vehicle follows
that lead vehicle. Along the route, the lead vehicle will perform a sudden
brake with negative velocities which cannot be seen during training. While
braking, we label the lead vehicle in the same way as we label content
anomalies.

Figure 6: Distribution of our content anomalies (left) and distribution of the
temporal anomalies (right), based on the 2D semantic labels in the dataset.

####  3.2.4 Data Generation.

As shown in Fig. 2, we provide sensor data for all positioned RGB cameras,
depth cameras, and lidar sensors. For regular perception tasks, we provide
panoptic masks for both camera and lidar. For each frame, also information
about the state of the ego vehicle is collected, such as its actions throttle,
street, and brake. For approaches that require additional information about
the planned route [81, 3, 44, 16], we provide a standard-format Bird’s-eye
view (BEV) representation of the planned route.

For the anomalous instances, we provide meta-data, such as their positions and
their size. Ground truth is embedded into the semantic masks for both camera
and lidar. As we designed the benchmark to compare methods that use different
modalities for the detection of anomalies, we represent all anomalies also in
a 3D voxel grid with customizable grid size. Based on our depth maps and lidar
point clouds, we fuse all visible points from all sensors in 3D and quantize
them into the voxel grid. In addition, the task of detecting anomalies in 3D
is especially helpful for downstream tasks, such as motion planning.

By using the CARLA simulation engine, we support the largest simulation
ecosystem in autonomous driving. However, it has inherent flaws, such as
imperfect behavior agents [47]. Thus, we perform minor data cleaning to remove
scenarios that contain collisions with pedestrians in the evaluation data and
scenarios that show high deceleration values for other vehicles in the
training data. This ensures that we avoid unlabeled anomalies in the
evaluation data and that our temporal anomalies are distinct from all
behaviors seen during training.

####  3.2.5 Labeling.

AnoVox provides 40 labels in the semantic masks, as shown in Fig. 7. We
provide labels for standard tasks as used by Cityscapes and CARLA [27, 30].
Additionally, we label the ego vehicle if visible. As described in Sec. 3.2.1,
some vehicles in the scene might switch from autopilot to a behavior agent. As
this driving behavior is not present in training data, we provide additional
labels for all vehicle classes while controlled by a behavior agent. Finally,
we provide labels for the included content and temporal anomalies. Labels for
the super-classes can be found in the semantic labels, while fine-granular,
individual anomaly labels are provided in additional metadata. Finally, voxels
are provided with an anomaly label when the closest point to their center is
anomalous.

road
---

sky
---

building
---

vegetation
---

ego vehicle
---

sidewalk
---

other
---

terrain
---

wall
---

road line
---

special
---

pole
---

fence
---

Car
---

agentcar
---

static
---

guard rail
---

ground
---

bridge
---

truck
---

animal
---

rail track
---

sudden brake
---

nature
---

airplane
---

falling
---

dynamic
---

bus
---

pedestrian
---

traffic light
---

traffic sign
---

home
---

agentbus
---

motorcycle
---

agenttruck
---

water
---

rider
---

bicycle
---

agentbicycle
---

agentmotorcycle
---

1M
---
10M
---
100M
---
1B
---
10B
---

Figure 7: Number of pixels per class: Bright green represents standard
classes, and dark green ones additional classes for vehicles that switched
into an agent behavior mode. The orange class shows our ego vehicle, and all
red classes represent anomalies.

###  3.3 Dataset and Statistics

Next to all code to generate further datasets, we provide over 1.3 TB of
generated data. We have split the data into chunks smaller than 50 GB for
easier processing, which we call instantiations. We provide 1,117 scenarios
based on the mono sensor configuration, 76 for the stereo configuration, and
35 for the multi configuration. While the stereo and multi configurations are
more of a symbolic nature, our dataset for the mono sensor configuration is
extensive. We provide data for eight different areas, and for each, we record
normality training data as well as evaluation data with content and temporal
anomalies, resulting in 24 different types of scenarios. Additionally, for
each of those 24, we create many scenarios where settings such as the weather,
time of day, or spawned anomaly vary. For content anomalies, 14.8% of all
frames contain visible anomalies in camera data and 74.8% in lidar data. For
temporal anomalies, 15.5% of all frames include anomalies equally visible in
both sensor modalities. As shown in Fig. 6 anomalies are well distributed over
the visible space but with a focus on critical areas in front of the ego
vehicle.

Finally, Fig. 7 gives an overview of the number of pixels per class label in
our dataset. We support all basic classes and provide additional classes for
our ego vehicle and other traffic participants that have switched from
autopilot to an agent behavior, so they can be excluded from evaluation if
needed, as they behave slightly differently than the ones available in the
training data.

Table 3: A comparison of current state-of-the-art methods for anomaly
detection.

| Year | Sensor | Aux
---|---|---|---
3DUIS [59] | 2022 | LiDAR | ✗
APF [50] | 2023 | LiDAR | ✗
cDNP [33] | 2023 | Camera | ✗
EAM [37] | 2023 | Camera | ✓
ElC-OIS [28] | 2023 | LiDAR | ✗
LS-VOS [63] | 2023 | LiDAR | ✗
Mask2Anomaly [64] | 2023 | Camera | ✓
Maskomaly [1] | 2023 | Camera | ✗
Maximized Entropy [24] | 2021 | Camera | ✓
Najibi et al. [56] | 2022 | LiDAR | ✗
ObsNet [5] | 2021 | Camera | ✗
RbA [57] | 2023 | Camera | Both
ReaL [22] | 2022 | LiDAR | Both
SynBoost [29] | 2021 | Camera | ✓

##  4 AnoVox Benchmark Suite

We provide a full evaluation suite for camera and lidar-based anomaly
detection methods for content anomalies. Based on pixel- or pointwise anomaly
scores, we voxelize the results and compare them against the voxelized anomaly
ground truth of AnoVox. We evaluate on voxel grid of size $100m\times
100m\times 64m$, where each voxel has a length of 0.5 m. We have performed
ablation studies on smaller voxel sizes, but have not found a significant
impact on the results.

###  4.1 Anomaly Detection

To get first insights into how current state-of-the-art methods for content
anomalies will perform on our benchmark, we have implemented two candidate
methods, one based on camera data and one based on lidar data. For their
selection, we have analyzed existing surveys and benchmark results [13, 15, 9,
23]. However, as there is no existing benchmark for lidar-based detection
method, this was more challenging. We performed an extensive search, also
focusing on open-set segmentation, as this is a similar but more active field.
Our final candidates can be found in Table 3. To emphasize the definition of
normality based on training data alone, we then neglected all methods that
require auxiliary data with anomalies during training. While we are also
interested in methods purely trained on raw data, we allowed to usage of
semantic labels during training, as the state-of-the-art heavily relies on
those. We then picked the best-performing method that provided open-source
code for implementations. For camera data, we selected the Rejected By All
(RbA) method from Nayal et al. [57], and for lidar data we selected the
Redundancy classifier (REAL) framework by Cen et al. [22]. While both methods
allow for the usage of auxiliary data during training, we did not fine-tune on
anomalies. For RbA this means we did not perform "Outlier Data Exposure" [57],
and for REAL this means we only used the "predictive distribution calibration"
without "unknown object synthesis" [22].

The Rejected by All model proposed by Nayal et al. uses camera data and is
based on the Mask2Former model [26]. The authors propose that object queries
are specialized on single classes, such that outliers can be detected when the
input is rejected by all queries. RbA was trained on Cityscapes and evaluated
on SegmentMeIfYouCan anomaly and obstacle tracks, as well as on the
Fishyscapes Lost&Found track, where they perform well with FPR95 values below
12.

The Redundancy classifier approach from Cen et al. uses lidar data and is
based on the Cylinder3D [83] framework. To learn high anomaly scores, the
authors introduce a new loss, where the second-highest predictions per point
are assigned to an unknown class, which they use for uncertainty prediction
during inference while also performing close-set semantic segmentation. RbA
was trained and evaluated on both SemanticKITTI [4] and Nuscenes [20]. The
approach works reasonably well with an AUPR of 10.0 and an AUROC of 77.5.
However, due to the domain gap, these values cannot easily be compared with
camera-based methods.

4.803149in2.401575inuse as bounding box,
clip0.000000ptcurrentstroke0.0000000pt0.000000in0.000000in4.803149in0.000000in4.803149in2.401575in0.000000in2.401575incurrentfill0.000000ptcurrentstroke0.0000000pt0.527895in0.782449in4.653149in0.782449in4.653149in2.230839in0.527895in2.230839infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt0.715407in0.782449in0.741817in0.782449in0.741817in0.901932in0.715407in0.901932infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt0.847457in0.782449in0.873867in0.782449in0.873867in2.062285in0.847457in2.062285infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt0.979508in0.782449in1.005918in0.782449in1.005918in1.021023in0.979508in1.021023infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.111558in0.782449in1.137968in0.782449in1.137968in1.123026in1.111558in1.123026infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.243609in0.782449in1.270019in0.782449in1.270019in0.950242in1.243609in0.950242infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.375659in0.782449in1.402069in0.782449in1.402069in0.808297in1.375659in0.808297infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.507709in0.782449in1.534119in0.782449in1.534119in0.848931in1.507709in0.848931infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.639760in0.782449in1.666170in0.782449in1.666170in0.785160in1.639760in0.785160infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.771810in0.782449in1.798220in0.782449in1.798220in0.784083in1.771810in0.784083infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.903860in0.782449in1.930271in0.782449in1.930271in1.295041in1.903860in1.295041infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.035911in0.782449in2.062321in0.782449in2.062321in0.873957in2.035911in0.873957infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.167961in0.782449in2.194371in0.782449in2.194371in2.027537in2.167961in2.027537infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.300012in0.782449in2.326422in0.782449in2.326422in0.783947in2.300012in0.783947infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.432062in0.782449in2.458472in0.782449in2.458472in0.787413in2.432062in0.787413infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.564112in0.782449in2.590522in0.782449in2.590522in0.949591in2.564112in0.949591infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.696163in0.782449in2.722573in0.782449in2.722573in0.877465in2.696163in0.877465infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.828213in0.782449in2.854623in0.782449in2.854623in0.819786in2.828213in0.819786infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.960264in0.782449in2.986674in0.782449in2.986674in0.786585in2.960264in0.786585infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.092314in0.782449in3.118724in0.782449in3.118724in0.783455in3.092314in0.783455infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.224364in0.782449in3.250774in0.782449in3.250774in0.798557in3.224364in0.798557infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.356415in0.782449in3.382825in0.782449in3.382825in0.787248in3.356415in0.787248infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.488465in0.782449in3.514875in0.782449in3.514875in0.997183in3.488465in0.997183infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.620515in0.782449in3.646926in0.782449in3.646926in0.785359in3.620515in0.785359infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.752566in0.782449in3.778976in0.782449in3.778976in0.820356in3.752566in0.820356infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.884616in0.782449in3.911026in0.782449in3.911026in0.807339in3.884616in0.807339infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.016667in0.782449in4.043077in0.782449in4.043077in0.808937in4.016667in0.808937infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.148717in0.782449in4.175127in0.782449in4.175127in0.813805in4.148717in0.813805infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.280767in0.782449in4.307177in0.782449in4.307177in0.839005in4.280767in0.839005infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.412818in0.782449in4.439228in0.782449in4.439228in0.782449in4.412818in0.782449infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt0.741817in0.782449in0.768227in0.782449in0.768227in0.782479in0.741817in0.782479infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt0.873867in0.782449in0.900278in0.782449in0.900278in2.161868in0.873867in2.161868infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.005918in0.782449in1.032328in0.782449in1.032328in1.041656in1.005918in1.041656infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.137968in0.782449in1.164378in0.782449in1.164378in1.473551in1.137968in1.473551infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.270019in0.782449in1.296429in0.782449in1.296429in0.845375in1.270019in0.845375infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.402069in0.782449in1.428479in0.782449in1.428479in0.802883in1.402069in0.802883infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.534119in0.782449in1.560529in0.782449in1.560529in0.821534in1.534119in0.821534infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.666170in0.782449in1.692580in0.782449in1.692580in0.787094in1.666170in0.787094infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.798220in0.782449in1.824630in0.782449in1.824630in0.786409in1.798220in0.786409infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt1.930271in0.782449in1.956681in0.782449in1.956681in1.453148in1.930271in1.453148infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.062321in0.782449in2.088731in0.782449in2.088731in0.876776in2.062321in0.876776infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.194371in0.782449in2.220781in0.782449in2.220781in1.803891in2.194371in1.803891infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.326422in0.782449in2.352832in0.782449in2.352832in0.782449in2.326422in0.782449infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.458472in0.782449in2.484882in0.782449in2.484882in0.782449in2.458472in0.782449infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.590522in0.782449in2.616933in0.782449in2.616933in0.994677in2.590522in0.994677infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.722573in0.782449in2.748983in0.782449in2.748983in0.782449in2.722573in0.782449infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.854623in0.782449in2.881033in0.782449in2.881033in0.782449in2.854623in0.782449infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt2.986674in0.782449in3.013084in0.782449in3.013084in0.783267in2.986674in0.783267infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.118724in0.782449in3.145134in0.782449in3.145134in0.782572in3.118724in0.782572infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.250774in0.782449in3.277184in0.782449in3.277184in0.810382in3.250774in0.810382infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.382825in0.782449in3.409235in0.782449in3.409235in0.785356in3.382825in0.785356infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.514875in0.782449in3.541285in0.782449in3.541285in0.957149in3.514875in0.957149infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.646926in0.782449in3.673336in0.782449in3.673336in0.783674in3.646926in0.783674infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.778976in0.782449in3.805386in0.782449in3.805386in0.835968in3.778976in0.835968infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt3.911026in0.782449in3.937436in0.782449in3.937436in0.795802in3.911026in0.795802infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.043077in0.782449in4.069487in0.782449in4.069487in0.786842in4.043077in0.786842infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.175127in0.782449in4.201537in0.782449in4.201537in0.804955in4.175127in0.804955infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.307177in0.782449in4.333588in0.782449in4.333588in0.805146in4.307177in0.805146infill0.527895in0.782449in4.125254in1.448390inclipcurrentfill0.000000ptcurrentstroke0.0000000pt4.439228in0.782449in4.465638in0.782449in4.465638in0.817746in4.439228in0.817746infillcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill0.741817in0.782449incurrentmarkertextcolortextcolor[x=0.450254in,
y=0.369112in, left,
base,rotate=45.000000]unlabeledcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill0.873867in0.782449incurrentmarkertextcolortextcolor[x=0.734054in,
y=0.520861in, left,
base,rotate=45.000000]roadcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.005918in0.782449incurrentmarkertextcolortextcolor[x=0.749138in,
y=0.403895in, left,
base,rotate=45.000000]sidewalkcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.137968in0.782449incurrentmarkertextcolortextcolor[x=0.888349in,
y=0.411056in, left,
base,rotate=45.000000]buildingcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.270019in0.782449incurrentmarkertextcolortextcolor[x=1.142140in,
y=0.532796in, left,
base,rotate=45.000000]wallcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.402069in0.782449incurrentmarkertextcolortextcolor[x=1.244182in,
y=0.502788in, left,
base,rotate=45.000000]fencecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.534119in0.782449incurrentmarkertextcolortextcolor[x=1.402831in,
y=0.529386in, left,
base,rotate=45.000000]polecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.666170in0.782449incurrentmarkertextcolortextcolor[x=1.319364in,
y=0.313869in, left, base,rotate=45.000000]traffic
lightcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.798220in0.782449incurrentmarkertextcolortextcolor[x=1.469147in,
y=0.331601in, left, base,rotate=45.000000]traffic
signcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill1.930271in0.782449incurrentmarkertextcolortextcolor[x=1.617225in,
y=0.347629in, left,
base,rotate=45.000000]vegetationcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.062321in0.782449incurrentmarkertextcolortextcolor[x=1.852942in,
y=0.451295in, left,
base,rotate=45.000000]terraincurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.194371in0.782449incurrentmarkertextcolortextcolor[x=2.087636in,
y=0.553939in, left,
base,rotate=45.000000]skycurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.326422in0.782449incurrentmarkertextcolortextcolor[x=2.011330in,
y=0.345582in, left,
base,rotate=45.000000]pedestriancurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.458472in0.782449incurrentmarkertextcolortextcolor[x=2.309111in,
y=0.511313in, left,
base,rotate=45.000000]ridercurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.590522in0.782449incurrentmarkertextcolortextcolor[x=2.473898in,
y=0.544050in, left,
base,rotate=45.000000]Carcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.722573in0.782449incurrentmarkertextcolortextcolor[x=2.559571in,
y=0.497673in, left,
base,rotate=45.000000]truckcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.854623in0.782449incurrentmarkertextcolortextcolor[x=2.744477in,
y=0.550529in, left,
base,rotate=45.000000]buscurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill2.986674in0.782449incurrentmarkertextcolortextcolor[x=2.658282in,
y=0.332283in, left,
base,rotate=45.000000]motorcyclecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.118724in0.782449incurrentmarkertextcolortextcolor[x=2.907640in,
y=0.449590in, left,
base,rotate=45.000000]bicyclecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.250774in0.782449incurrentmarkertextcolortextcolor[x=3.077883in,
y=0.487783in, left,
base,rotate=45.000000]staticcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.382825in0.782449incurrentmarkertextcolortextcolor[x=3.127409in,
y=0.405259in, left,
base,rotate=45.000000]dynamiccurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.514875in0.782449incurrentmarkertextcolortextcolor[x=3.351873in,
y=0.497672in, left,
base,rotate=45.000000]othercurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.646926in0.782449incurrentmarkertextcolortextcolor[x=3.475399in,
y=0.489148in, left,
base,rotate=45.000000]watercurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.778976in0.782449incurrentmarkertextcolortextcolor[x=3.509579in,
y=0.391277in, left, base,rotate=45.000000]road
linecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill3.911026in0.782449incurrentmarkertextcolortextcolor[x=3.697555in,
y=0.447203in, left,
base,rotate=45.000000]groundcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill4.043077in0.782449incurrentmarkertextcolortextcolor[x=3.850066in,
y=0.467663in, left,
base,rotate=45.000000]bridgecurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill4.175127in0.782449incurrentmarkertextcolortextcolor[x=3.892772in,
y=0.378320in, left, base,rotate=45.000000]rail
trackcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill4.307177in0.782449incurrentmarkertextcolortextcolor[x=4.007772in,
y=0.361269in, left, base,rotate=45.000000]guard
railcurrentfill0.803000ptcurrentstroke0ptcurrentmarker0.000000in-0.048611in0.000000in0.000000in0.000000in0.000000in0.000000in-0.048611instroke,fill4.439228in0.782449incurrentmarkertextcolortextcolor[x=4.183812in,
y=0.405258in, left,
base,rotate=45.000000]anomalytextcolortextcolor[x=2.590522in,y=0.248766in,,top]Classcurrentfill0.803000ptcurrentstroke0ptcurrentmarker-0.048611in0.000000in-0.000000in0.000000in-0.000000in0.000000in-0.048611in0.000000instroke,fill0.527895in0.782449incurrentmarkertextcolortextcolor[x=0.304321in,
y=0.758336in, left,
base]$\displaystyle{0.0}$currentfill0.803000ptcurrentstroke0ptcurrentmarker-0.048611in0.000000in-0.000000in0.000000in-0.000000in0.000000in-0.048611in0.000000instroke,fill0.527895in1.264346incurrentmarkertextcolortextcolor[x=0.304321in,
y=1.240234in, left,
base]$\displaystyle{0.1}$currentfill0.803000ptcurrentstroke0ptcurrentmarker-0.048611in0.000000in-0.000000in0.000000in-0.000000in0.000000in-0.048611in0.000000instroke,fill0.527895in1.746244incurrentmarkertextcolortextcolor[x=0.304321in,
y=1.722131in, left,
base]$\displaystyle{0.2}$currentfill0.803000ptcurrentstroke0ptcurrentmarker-0.048611in0.000000in-0.000000in0.000000in-0.000000in0.000000in-0.048611in0.000000instroke,fill0.527895in2.228141incurrentmarkertextcolortextcolor[x=0.304321in,
y=2.204029in, left,
base]$\displaystyle{0.3}$textcolortextcolor[x=0.248766in,y=1.506644in,,bottom,rotate=90.000000]Frequency0.803000ptcurrentstroke0pt0.527895in0.782449in0.527895in2.230839instroke0.803000ptcurrentstroke0pt4.653149in0.782449in4.653149in2.230839instroke0.803000ptcurrentstroke0pt0.527895in0.782449in4.653149in0.782449instroke0.803000ptcurrentstroke0pt0.527895in2.230839in4.653149in2.230839instrokecurrentfill0.8000001.003750ptcurrentstroke0.8000000pt4.087026in1.931765in4.594816in1.931765in4.611483in1.931765in4.611483in1.948431in4.611483in2.172505in4.611483in2.189172in4.594816in2.189172in4.087026in2.189172in4.070359in2.189172in4.070359in2.172505in4.070359in1.948431in4.070359in1.931765in4.087026in1.931765instroke,fillcurrentfill0.000000ptcurrentstroke0.0000000pt4.103692in2.097505in4.270359in2.097505in4.270359in2.155839in4.103692in2.155839infilltextcolortextcolor[x=4.337026in,y=2.097505in,left,base]Traincurrentfill0.000000ptcurrentstroke0.0000000pt4.103692in1.981302in4.270359in1.981302in4.270359in2.039635in4.103692in2.039635infilltextcolortextcolor[x=4.337026in,y=1.981302in,left,base]Eval

Figure 8: Class distributions for the train and evaluation datasets.

####  4.1.1 Training

For a fair comparison, we have trained both methods on the same training
dataset. For this, we have created a small normality instantiation in the size
of the Cityscapes dataset that consists of 2,975 frames with temporal
scenarios. Contrary to the setting of REAL, where unknown objects are included
in the training set but ignored for the loss computation, our training dataset
does not include any anomalies. We followed the standard training procedures
and parameters as provided by the authors.

Table 4: Evaluation of REAL and RbA on AnoVox.

|  AUPR $\uparrow$ | AUROC$\uparrow$ | F1$\uparrow$ | PPV$\uparrow$ | FPR95$\downarrow$
---|---|---|---|---|---
REAL | 0.14 | 43.30 | 0.0 | 0.0 | 100
REAL+norm | 0.04 | 43.53 | 0.0 | 0.0 | 100
REALbig | 0.17 | 44.7 | 0.0 | 0.0 | 100
REALmedium | 0.11 | 42.8 | 0.0 | 0.0 | 100
REALsmall | 0.21 | 55.1 | 0.0 | 0.0 | 100
RbA | 0.7 | 57.3 | 2.6 | 1.4 | 100
RbA+norm | 0.2 | 57.6 | 2.6 | 0.4 | 100
RbAbig | 2.3 | 54.9 | 3.7 | 2.7 | 100
RbAmedium | 0.6 | 60.5 | 0.0 | 0.0 | 100
RbAsmall | 0.01 | 53.2 | 0.0 | 0.0 | 100

####  4.1.2 Evaluation

For the evaluation, we created another small instantiation with content
anomalies. The class distribution between the training and evaluation datasets
can be found in Figure 8. To evaluate both methods, their anomaly scores need
to be mapped to voxel space first. While this is straightforward for lidar
data, we lifted the anomaly scores from RbA into 3D using ground truth depth
data. Due to this evaluation in voxel space, it must be noted that the class
imbalance between normal data and anomalies is much larger than in the sensor
space due to quantization effects. Thus, results cannot be compared to
reported values from the state of the Art. However, we have also performed
evaluations on the sensor data directly, confirming higher scores.

As shown in Table 4, both methods had significant issues detecting anomalies
in our setting. While we observed stable training performance and saw improved
performance when trained on larger datasets, the results were rather
surprising. RbA was able to detect some anomalies successfully, as shown in
Fig. 1, but struggled with most. We were able to improve the performance with
more training data, but not substantially. REAL on the other hand, while
generating well-performing closed-set predictions, was unable to generate any
meaningful uncertainties in the absence of unknown objects in the training
data, as exemplarily shown in Fig. 9.

This shows that the task of anomaly detection becomes much harder on a
challenging benchmark such as AnoVox, where anomalies are not defined by not
belonging to Cityscapes classes, but are defined by being absent from the
training set, as most state-of-the-art methods currently utilize deviations
based on semantic segmentations to derive anomalies. In addition, there is no
domain shift for the anomaly classes, as they are rendered in the same
simulation engine from which the training data stems from. Finally, lots of
frames without visible anomalies and particularly small anomalies raise the
bar additionally. This setting, combined with the induced class imbalance due
to our quantization loss during voxelization, makes it much harder to perform
well on AnoVox compared to existing benchmarks, which we will hope benefit the
community greatly in the upcoming years.

##  5 Conclusion

In this work, we presented the AnoVox benchmark for ANOmaly detection in
autonomous driving. We are the first benchmark to provide ground truth in the
form of semantic masks for RGB images and lidar point clouds as well as
spatial VOXels, allowing us to compare methods based on arbitrary sensor
modalities.

Figure 9: AnoVox scene with a cow as an animal content anomaly (left). The RbA
(middle) anomaly detection method is only able to detect some border parts as
anomalous, while REAL (right) assigns the same uncertainty to the cow as it
does to the ground. In the accompanying closed-set detection, it is mostly
classified as a car.

We systematically analyzed the shortcomings of existing benchmarks, most
notably no clear definition of normality and a strong focus on RGB data.
AnoVox is the first and largest benchmark that provides a concise
formalization of normality, accompanied by training data that complies with
that definition. Finally, we provided first insights on how well current
state-of-the-art models perform on this new paradigm of anomaly detection
benchmarking. We see a great need for anomaly detection methods to learn
normality based on training data instead of detecting deviations from known
Cityscape classes.

Limitations. At the moment, each voxel contains only a single class. This can
lead to cases where small anomalies are not represented in single frames of
voxelized data. For our evaluation, we map camera-based detection into 3D by
using ground truth depth. We have performed experiments using ZoeDepth [7] to
map these predictions into 3D, but the overlap between the resulting voxels
and the ground truth was too large. Here, more sophisticated methods are
necessary. We limited ourselves to only a single temporal anomaly, more types
could be implemented. Finally, while simulation is necessary to have full
control over both training and evaluation, this leads to a Sim-2-Real gap.

Potential negative impact. As we provide a large number of anomalies, others
might use the dataset for extensive fine-tuning. This can lead to cases where
anomaly detection methods perform extremely well on benchmarks but might still
fail in open-world settings. Thus, we do not recommend using our data with
anomalies for anything else than benchmarking.

Outlook. We plan to implement multi-class voxel in a future release to reduce
losses during quantization. The Sim-2-Real gap could be tackled by style
transfers, as proposed by Tian et al. [69]. We would like to include radar
data in a future release, but the current implementation in CARLA is too naive
to be helpful. Finally, we plan to release more data with our surround sensor
setup.

## References

  * [1] Ackermann, J., Sakaridis, C., Yu, F.: Maskomaly: Zero-Shot Mask Anomaly Segmentation. In: BMVC (2023)
  * [2] Aliakbarian, M.S., Saleh, F.S., Salzmann, M., Fernando, B., Petersson, L., Andersson, L.: Viena2: A driving anticipation dataset. In: ACCV (2018)
  * [3] Bansal, M., Krizhevsky, A., Ogale, A.: Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In: RSS (2019)
  * [4] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., Gall, J.: SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences. In: ICCV (2019)
  * [5] Besnier, V., Bursuc, A., Picard, D., Briot, A.: Triggering failures: Out-of-distribution detection by learning from local adversarial attacks in semantic segmentation. In: ICCV (2021)
  * [6] Bevandić, P., Krešo, I., Oršić, M., Šegvić, S.: Simultaneous Semantic Segmentation and Outlier Detection in Presence of Domain Shift. In: GCPR (2019)
  * [7] Bhat, S.F., Birkl, R., Wofk, D., Wonka, P., Müller, M.: ZoeDepth: Zero-shot transfer by combining relative and metric depth. arXiv:2302.12288 (2023)
  * [8] Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving. In: ICCVW (2019)
  * [9] Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation. IJCV 129 (2021)
  * [10] Bogdoll, D., Breitenstein, J., Heidecker, F., Bieshaar, M., Sick, B., Fingscheidt, T., Zöllner, J.M.: Description of Corner Cases in Automated Driving: Goals and Challenges. In: ICCVW (2021)
  * [11] Bogdoll, D., Guneshka, S., Zöllner, J.M.: One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Driving. In: ECCVW (2022)
  * [12] Bogdoll, D., Nekolla, M., Joseph, T., Zöllner, J.M.: Quantification of Actual Road User Behavior on the Basis of Given Traffic Rules. In: IEEE IV (2022)
  * [13] Bogdoll, D., Nitsche, M., Zöllner, J.M.: Anomaly Detection in Autonomous Driving: A Survey. In: CVPRW (2022)
  * [14] Bogdoll, D., Schreyer, F., Zöllner, J.M.: AD-Datasets: A Meta-Collection of Data Sets for Autonomous Driving. In: VEHITS (2022)
  * [15] Bogdoll, D., Uhlemeyer, S., Kowol, K., Zöllner, J.M.: Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey. In: IEEE IV (2023)
  * [16] Bogdoll, D., Yang, Y., Zöllner, J.M.: MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations. arXiv:2311.11762 (2023)
  * [17] Breitenstein, J., Termohlen, J.A., Lipinski, D., Fingscheidt, T.: Corner Cases for Visual Perception in Automated Driving: Some Guidance on Detection Approaches. arXiv:2102.05897 (2021)
  * [18] Breitenstein, J., Termöhlen, J.A., Lipinski, D., Fingscheidt, T.: Systematization of Corner Cases for Visual Perception in Automated Driving. In: IEEE IV (2020)
  * [19] Bu, T., Zhang, X., Mertz, C., Dolan, J.M.: CARLA Simulated Data for Rare Road Object Detection. In: IEEE ITSC (2021)
  * [20] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuScenes: A Multimodal Dataset for Autonomous Driving. In: CVPR (2020)
  * [21] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV (2021)
  * [22] Cen, J., Yun, P., Zhang, S., Cai, J., Luan, D., Wang, M.Y., Liu, M., Tang, M.: Open-world semantic segmentation for LIDAR point clouds. In: ECCV (2022)
  * [23] Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., Rottmann, M.: SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation. In: NeurIPS (2021)
  * [24] Chan, R., Rottmann, M., Gottschalk, H.: Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In: ICCV (2021)
  * [25] Chan, R., Uhlemeyer, S., Rottmann, M., Gottschalk, H.: Deep Neural Networks and Data for Automated Driving, chap. Detecting and Learning the Unknown in Semantic Segmentation. Springer (2022)
  * [26] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR (2022)
  * [27] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
  * [28] Deng, W., Huang, K., Yu, Q., Lu, H., Zheng, Z., Chen, X.: ElC-OIS: Ellipsoidal Clustering for Open-World Instance Segmentation on LiDAR Data. In: IROS (2023)
  * [29] Di Biase, G., Blum, H., Siegwart, R., Cadena, C.: Pixel-wise anomaly detection in complex driving scenes. In: CVPR (2021)
  * [30] Dosovitskiy, A., Ros, G., Codevilla, F., López, A.M., Koltun, V.: Carla: An open urban driving simulator. In: CoRL (2017)
  * [31] Du, X., Wang, X., Gozum, G., Li, Y.: Unknown-Aware Object Detection: Learning What You Don’t Know from Videos in the Wild. In: CVPR (2022)
  * [32] Franchi, G., Yu, X., Bursuc, A., Tena, A., Kazmierczak, R., Dubuisson, S., Aldea, E., Filliat, D.: Muad: Multiple uncertainties for autonomous driving benchmark for multiple uncertainty types and tasks. In: BMVC (2022)
  * [33] Galesso, S., Argus, M., Brox, T.: Far away in the deep space: Dense nearest-neighbor-based out-of-distribution detection. In: ICCVW (2023)
  * [34] Gong, L., Zhang, Y., Xia, Y., Zhang, Y., Ji, J.: SDAC: A Multimodal Synthetic Dataset for Anomaly and Corner Case Detection in Autonomous Driving. AAAI 38(3) (2024)
  * [35] Grcić, M., Bevandić, P., Kalafatić, Z., Šegvić, S.: Dense out-of-distribution detection by robust learning on synthetic negative data. Sensors 24 (2024)
  * [36] Grcić., M., Bevandić., P., Segvić., S.: Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. In: Int. Conf. Comp. Vis., Imag. Comp. Graph. Theo. Appl. (2021)
  * [37] Grcić, M., Šarić, J., Šegvić, S.: On advantages of mask-level recognition for outlier-aware segmentation. In: CVPRW (2023)
  * [38] Hanselmann, N., Renz, K., Chitta, K., Bhattacharyya, A., Geiger, A.: KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients. In: ECCV (2022)
  * [39] Heidecker, F., Bieshaar, M., Sick, B.: Corner cases in machine learning processes. AI Perspectives & Advances 6(1) (2024)
  * [40] Heidecker, F., Breitenstein, J., Rösch, K., Löhdefink, J., Bieshaar, M., Stiller, C., Fingscheidt, T., Sick, B.: An Application-Driven Conceptualization of Corner Cases for Perception in Highly Automated Driving. In: IEEE IV (2021)
  * [41] Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Kwon, J., Mostajabi, M., Steinhardt, J., Song, D.: Scaling Out-of-Distribution Detection for Real-World Settings. In: ICML (2022)
  * [42] Hendrycks, D., Gimpel, K.: A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In: ICLR (2016)
  * [43] Hess, G., Jaxing, J., Svensson, E., Hagerman, D., Petersson, C., Svensson, L.: Masked autoencoder for self-supervised pre-training on lidar point clouds. In: IEEE Winter Conf. on Appl. of Comp. Vis. (2023)
  * [44] Hu, A., Corrado, G., Griffiths, N., Murez, Z., Gurau, C., Yeo, H., Kendall, A., Cipolla, R., Shotton, J.: Model-based imitation learning for urban driving. In: NeurIPS (2022)
  * [45] Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: GAIA-1: A Generative World Model for Autonomous Driving. arXiv:2309.17080 (2023)
  * [46] Kim, H., Lee, K., Hwang, G., Suh, C.: Crash to Not Crash: Learn to Identify Dangerous Vehicles Using a Simulator. AAAI 33 (2019)
  * [47] Kim, S.: Multiple issues in the behavior agent. https://github.com/carla-simulator/carla/issues/5398 (2022), accessed: 2024-03-07
  * [48] Kösel, M., Schreiber, M., Ulrich, M., Gläser, C., Dietmayer, K.: Revisiting Out-of-Distribution Detection in LiDAR-based 3D Object Detection. arXiv:2404.15879 (2024)
  * [49] Li, H., Li, Y., Wang, H., Zeng, J., Xu, H., Cai, P., Chen, L., Yan, J., Xu, F., Xiong, L., Wang, J., Zhu, F., Yan, K., Xu, C., Wang, T., Xia, F., Mu, B., Peng, Z., Lin, D., Qiao, Y.: Open-sourced data ecosystem in autonomous driving: the present and future. arXiv:2312.03408 (2024)
  * [50] Li, J., Dong, Q.: Open-Set Semantic Segmentation for Point Clouds via Adversarial Prototype Framework. In: CVPR (2023)
  * [51] Li, K., Chen, K., Wang, H., Hong, L., Ye, C., Han, J., Chen, Y., Zhang, W., Xu, C., Yeung, D.Y., Liang, X., Li, Z., Xu, H.: CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving. In: ECCV (2022)
  * [52] Li, Y., Zhang, W., Chen, K., Liu, Y., Li, P., Gao, R., Hong, L., Tian, M., Zhao, X., Li, Z., Yeung, D.Y., Lu, H., Jia, X.: Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases. arXiv:2404.10595 (2024)
  * [53] Liu, M., Yurtsever, E., Zhou, X., Fossaert, J., Cui, Y., Zagar, B.L., Knoll, A.C.: A survey on autonomous driving datasets: Data statistic, annotation, and outlook. arXiv:2401.01454 (2024)
  * [54] Maag, K., Chan, R., Uhlemeyer, S., Kowol, K., Gottschalk, H.: Two Video Data Sets for Tracking and Retrieval of Out of Distribution Objects. In: ACCV (2023)
  * [55] Mohan, R., Kumaraswamy, K., Hurtado, J.V., Petek, K., Valada, A.: Panoptic out-of-distribution segmentation. arXiv:2310.11797 (2023)
  * [56] Najibi, M., Ji, J., Zhou, Y., Qi, C.R., Yan, X., Ettinger, S., Anguelov, D.: Motion inspired unsupervised perception and prediction in autonomous driving. In: ECCV (2022)
  * [57] Nayal, N., Yavuz, M., Henriques, J.F., Güney, F.: RbA: Segmenting unknown regions rejected by all. In: ICCV (2023)
  * [58] Nunes, I., Laranjeira, C., Oliveira, H., dos Santos, J.A.: A systematic review on open-set segmentation. Computers & Graphics 115 (2023)
  * [59] Nunes, L., Chen, X., Marcuzzi, R., Osep, A., Leal-Taixé, L., Stachniss, C., Behley, J.: Unsupervised Class-Agnostic Instance Segmentation of 3D LiDAR Data for Autonomous Vehicles. IEEE Rob. and Auto. Letters 7(4) (2022)
  * [60] Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: DINOv2: Learning Robust Visual Features without Supervision. Trans. on Mach. Learn. Reas. (2023)
  * [61] Pfeil, J., Wieland, J., Michalke, T., Theissler, A.: On Why the System Makes the Corner Case: AI-based Holistic Anomaly Detection for Autonomous Driving. In: IEEE IV (2022)
  * [62] Pinggera, P., Ramos, S., Gehrig, S., Franke, U., Rother, C., Mester, R.: Lost and Found: Detecting Small Road Hazards for Self-Driving Vehicles. In: IROS (2016)
  * [63] Piroli, A., Dallabetta, V., Kopp, J., Walessa, M., Meissner, D., Dietmayer, K.: LS-VOS: Identifying outliers in 3d object detections using latent space virtual outlier synthesis. IEEE ITSC (2023)
  * [64] Rai, S.N., Cermelli, F., Fontanel, D., Masone, C., Caputo, B.: Unmasking anomalies in road-scene segmentation. In: ICCV (2023)
  * [65] Rempe, D., Philion, J., Guibas, L.J., Fidler, S., Litany, O.: Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior. In: CVPR (2022)
  * [66] Rösch, K., Heidecker, F., Truetsch, J., Kowol, K., Schicktanz, C., Bieshaar, M., Sick, B., Stiller, C.: Space, Time, and Interaction: A Taxonomy of Corner Cases in Trajectory Datasets for Automated Driving. In: Symp. Series Comp. Intell. (2022)
  * [67] Sakaridis, C., Dai, D., Van Gool, L.: ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding. In: ICCV (2021)
  * [68] Sena Ferreira, R., Guérin, J., Guiochet, J., Waeselynck, H.: SiMOOD: Evolutionary Testing Simulation with Out-Of-Distribution Images. In: Pac. Rim Int. Symp Dep. Comp. (2022)
  * [69] Tian, B., ang Gao, H., Cui, L., Zheng, Y., Luo, L., Wang, B., Zhi, R., Zhou, G., Zhao, H.: Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics. arXiv:2401.04942 (2024)
  * [70] Tremblay, M., Halder, S.S., de Charette, R., Lalonde, J.F.: Rain Rendering for Evaluating and Improving Robustness to Bad Weather. IJCV 129 (2021)
  * [71] Urtasun, R.: Interpretable neural motion planner. https://youtube.com/watch?v=PSZ2Px9PrHg (2021), accessed: 2024-03-01
  * [72] Wang, T., Kim, S., Ji, W., Xie, E., Ge, C., Chen, J., Li, Z., Luo, P.: DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving. In: AAAI (2024)
  * [73] Waymo: Waymo One: The next step on our self-driving journey. https://waymo.com/blog/2018/10/where-next-10-million-miles-will-take-us/ (2018), accessed: 2024-02-22
  * [74] Waymo: Advice Letter 0002 (Tier 2). https://cpuc.ca.gov/-/media/cpuc-website/divisions/consumer-protection-and-enforcement-division/documents/tlab/av-programs/waymo-llc-cpuc-advice-letter-0002-tier-2--january-2024-passenger-safety-plan-update-january-192024.pdf (2024), accessed: 2024-02-22
  * [75] Waymo: Waymo One. https://waymo.com/waymo-one/ (2024), accessed: 2024-02-22
  * [76] Wiederer, J., Schmidt, J., Kressel, U., Dietmayer, K., Belagiannis, V.: A benchmark for unsupervised anomaly detection in multi-agent trajectories. In: IEEE ITSC (2022)
  * [77] Xu, S., Gilpin, L.H.: DANGER: A Framework of Danger-Aware Novel Dataset Generator Extension for Robustness Test of Machine Learning. In: BayLearn Mach. Learn. Symp. (2022)
  * [78] Yang, H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H., Qiu, Q., Lin, B., He, X., Ouyang, W.: Unipad: A universal pre-training paradigm for autonomous driving. In: CVPR (2024)
  * [79] Zendel, O., Honauer, K., Murschitz, M., Steininger, D., Dominguez, G.F.: WildDash - Creating Hazard-Aware Benchmarks. In: ECCV (2018)
  * [80] Zhang, L., Xiong, Y., Yang, Z., Casas, S., Hu, R., Urtasun, R.: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion. In: ICLR (2024)
  * [81] Zhang, Z., Liniger, A., Dai, D., Yu, F., Van Gool, L.: End-to-end urban driving by imitating a reinforcement learning coach. In: ICCV (2021)
  * [82] Zhou, J., Beyerer, J.: Corner cases in data-driven automated driving: Definitions, properties and solutions. In: IEEE IV (2023)
  * [83] Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., Lin, D.: Cylindrical and asymmetrical 3d convolution networks for LiDAR segmentation. In: CVPR (2021)